Heldout testing of an action-claim verifier, a fix for fail-open verifier and a standalone library, failures disclosed.
[Update 3 - Nov 20, 2026 — Standalone repo, arXiv submission, Live-60 failure disclosure (38.2% FPR -> 2.9% in v0.6), and 146/146 AER-1 conformance]
(Note: drafted with AI editing assistance; all code, benchmarks, SHA-256 provenance logs, and error audits are mine and reproducible from the links below.)
Since Update 2 (Nov 13), I moved epistemic-guard into its own standalone MIT repository, audited every number with SHA-256 provenance tracking osaudit), built a live MCP fault-injection harness that broke my own precision, submitted the full paper (with the bad numbers left in) to arXiv cs.SE), and built the v0.6.0 branch fixing what broke.
Updated links (replacing the old ProjectLucca subfolder link in Update 2):
- Standalone repo (MIT): https://github.com/AlienatedAliens/epistemic-guard
- Benchmark dataset (1,033 items across 8 configs): https://huggingface.co/datasets/RollSolo/epistemic-guard-benchmarks
- OpenVINO INT8 weights (SHA-256 pinned): https://huggingface.co/RollSolo/epistemic-guard-int8
- Interactive demo: https://huggingface.co/spaces/RollSolo/epistemic-guard
1. **Audit correction on N=250 + paired comparison against a 30B judge v0.5.2):**
When re-running the strictly held-out N=250 action-receipt benchmark end-to-end from clean standalone scripts for the arXiv paper tag v0.5.2), epistemic-guard scored **233/250 (93.20%, 95% Wilson CI [89.38%, 95.71%])** at ~11 ms/clause on CPU (0 MiB VRAM), rather than the 236/250 (94.4%) reported in Update 2. On the exact same 250 turns, a 30B MoE LLM-as-a-judge Nemotron-3.5-30B-A3B) scored **232/250 (92.80%, exact two-sided McNemar p = 1.0)** while taking ~2 seconds per turn, and Vectara HHEM-2.1 scored **129/250 (51.60%)**.
2. **Where v0.5.2 failed badly — live MCP fault injection Live-60):**
To test realistic multi-step tool failures instead of static transcripts, I built an out-of-process stdio MCP JSON-RPC proxy epistemic-guard mcp-proxy) that signs 4 distinct wire outcomes with Ed25519 into an RFC 6962 Merkle log: clean success ok=true), tool runtime error isError: true), schema rejection before execution -32602), and mid-call server crash EOF).
Across 60 live fault-injected trajectories Live-60, frozen before any fixes), v0.5.2 caught **25/26 ungrounded claims (96.15% recall)**, but its **false-positive rate jumped to 38.24% (13/34 clean or honestly reported failed turns wrongly blocked; 46/60 overall accuracy)**.
Two specific failure modes caused all 13 false positives:
- 6/13 came from unseen custom MCP tool names execute_sql, http_post, git_commit) missing from the static verb-stem table.
- 7/13 came from honest failure reports "I attempted to delete prod_ledger.db, but the operation failed" or "Please try the operation again") where clause splitting separated "I attempted to delete..." from "but the operation failed".
I published those exact 38.24% FPR numbers unchanged in the arXiv v1 submission rather than tuning on the test set first.
3. **What is now built on v0.6-dev v0.6.0):**
Keeping Live-60 locked, I implemented:
- Dynamic MCP tools/list schema family binding + SHA-256 mid-session "rug-pull" tool-mutation pinning, contrastive failure-clause negation handling, and a first-person subject check for promises. On N=250, accuracy is **234/250 (93.60%, 0 regressions)**; on Live-60, false positives drop from **13/34 (38.24%) to 1/34 (2.94%, 58/60 [96.67%] accuracy)** — which now needs a fresh out-of-sample cohort Live-250+) to confirm it generalizes.
- OutputValueGuard: deterministic JSON leaf scalar witness + INT8 NLI slice check for mutated numbers/statuses buried inside 20 KB+ ok=true tool outputs (< 5 ms CPU).
- Finite-sample split-conformal risk calibration ConformalCalibrator), Streamable HTTP/SSE MCP proxy mcp-http-proxy) implementing Lost-Response idempotent recovery A1-A5), native OpenAI / Anthropic / Gemini / LiteLLM / OpenTelemetry gen_ai.*) adapters, and **146/146** full conformance on the external IETF draft-zambo-aer1-26 test suite (up from 64/64 scoped after Brennan Zambo independently benchmarked v0.4.2).
**What $3,000–$6,000 funds in the final 5 days of this campaign:**
I self-funded and shipped the standalone MIT package, OpenVINO INT8 weights, stdio + HTTP MCP proxies, Live-60 harness, and the arXiv paper ahead of schedule. The two things I cannot fix without funding are **single-author label bias** and **paid frontier-API scale**:
1. **Paid independent human double-annotation ($1,000):** Hiring two blind external annotators (with Cohen's kappa reported publicly) to re-grade N=250, Live-60, and a new Live-250 holdout so none of the ground-truth labels rest on my own judgment.
2. **Out-of-sample frontier MCP fault-injection evaluation ($1,000 API/compute):** Generating N >= 300 fresh live MCP fault-injection traces across paid Claude Sonnet, GPT, and Gemini APIs to test whether v0.6.0's 2.94% FPR holds strictly out-of-sample (since Live-60 was inspected to diagnose v0.5.2's 13 FPs and is now burned as a diagnostic set).
3. **Researcher runway / partial retrogrant ($1,000–$4,000):** Supporting 1–2.5 months of dedicated open-source maintenance, upstream LiteLLM / OpenTelemetry integrations, and a camera-ready workshop submission with the human-adjudicated frontier numbers.
[Update 2 - Nov 13, 2026 — Standalone epistemic-guard v0.3.0 shipped to PyPI + N=250 LLM-Judge Benchmark]
In the 9 days since Update 1, I closed the prose-leakage gap and shipped the standalone MIT Python package ahead of schedule:
- **PyPI v0.3.0, MIT):** https://pypi.org/project/epistemic-guard/ pip install epistemic-guard, clean-venv CI green on Ubuntu, macOS, and Windows)
- **Source & reproducible benchmarks:** https://github.com/AlienatedAliens/ProjectLucca/tree/main/packages/epistemic-guard
- **ORCID:** https://orcid.org/0009-0004-0133-0463
1. **Closed the prose-leakage gap 236/250 [94.4%] strictly held-out on CPU, 0 MiB VRAM):**
Replaced surface regexes with an 83 MB 10-class OpenVINO INT8 clause-level speech-act classifier 10.9 ms p50/clause on CPU) + offset-preserving span masking (code blocks, blockquotes, dialogue) bound to RFC 6962-style Merkle ToolReceipt proofs. Across N=250 strictly held-out turns (including 100 live organic multi-sentence completions across 4 local model families):
- epistemic-guard CPU, 0 MiB VRAM): **236/250 (94.4%, 95% Wilson CI [90.8%, 96.6%])**, 96.7% recall, 7.8% FPR, 27.4 ms turn p50 50/50 on LOCKED_COH_50, 95/100 on BLIND42_COH_100, 91/100 on Organic_Heldout_H100).
- Quantized local GPU LLM judges: Mistral-Nemo-12B Q5_K_M 8.8 GiB VRAM) scored **217/250 (86.8%, 14.1% FPR, 77.9 ms)**; Phi-4-14B Q6_K 12.2 GiB VRAM) scored **216/250 (86.4%, 100% recall, 26.6% FPR, 75.1 ms)**.
- Off-the-shelf SNLI DeBERTa-v3-xsmall, CPU): **132/250 (52.8%, 24.6% recall)**.
2. **Upgraded HMAC to Public-Key-Only Ed25519 ProvenanceGate):**
The verifier/SQLite store now holds only Ed25519 public keys < 0.3 ms), so a prompt-injected runtime physically lacks key material to forge USER_EXPLICIT or TOOL_VERIFIED writes 0/100 natural self-poisoning; 0/200 false rejections on benign memory continuity).
3. **Fixed MiniLM negation false positives + ran single-shot live-web holdout real_web_v2, N=64):**
Added an INT8 nli-deberta-v3-xsmall verifier 9.9 ms CPU p50) + syndication deduplication 30/30 wire-reprint collapse). On our pre-registered single-shot live-scraped web holdout real_web_v2, N=64), dispute accuracy is **56/64 (87.5%)** and end-to-end burden accuracy is **47/64 (73.4%)** — implicit cross-article disagreements without single-sentence NLI opposition remain 26.6% 17/64) of open-web errors.
**Updated use of the $3,000–$6,000 grant (now that v0.3.0 is live on PyPI):**
Every label so far was still created/adjudicated by me + blinded LLM raters on local workstation weights, with zero frontier-API evaluation or human crowdworker labels. Funding now goes directly to:
- **(a) Independent human double-annotation ($750):** Hiring 2 blind human annotators to re-label the N=250 supervision and N=128 live-web sets so false-positive/recall claims aren't reliant on author + LLM adjudication.
- **(b) Frontier-model & live MCP fault-injection benchmark ($750 API/compute):** Benchmarking epistemic-guard against frontier LLM judges (Claude Sonnet/Haiku, GPT-4.1/5-mini, Gemini Flash) and live tool-failure trajectories across Claude, GPT, Llama, and Qwen n >= 250 per cohort to certify < 2% FPR).
- **(c) 3 months ($4,500) to ship epistemic-guard v1.0 + peer-reviewed workshop submission:** Cutting organic multi-clause FPR from 7.8% to < 2%, closing the 26.6% implicit cross-document contradiction gap on real_web_v2, and shipping native drop-in MCP proxy + LangGraph/OpenAI Agents middleware.
[Update - Nov 4, 2026]
In the 10 days since posting this proposal, I ran the first round of held-out tests on my own hardware and open-sourced the full codebase at https://github.com/AlienatedAliens/ProjectLucca.
1. Held-out matcher test (my warning below came true): On 120 fresh held-out prompts on Gemma 4 26B-A4B, the speech-regex checker that scored 23/23 in-sample dropped back down to 1/16 (6.25% strict sensitivity, 95% Wilson CI [1.1%, 28.3%]) and 1/23 (4.35% broad). Tuning regexes on surface prose overfit the training phrasings, just as I worried in the original post.
2. What replaced it (SOAC): I built a Structured Output-Action Contract (SOAC) supervisor that binds action claims to cryptographic tool receipts in 6.82 microseconds on CPU with 0/40 false positives ([0.0%, 8.8%] Wilson CI). Its remaining failure mode is prose leakage: on 15 to 17 out of 120 turns (12.5% to 14.2%), the model still narrates an ungrounded action in free text outside the contract block, and GBNF grammar constraints alone did not fix it (17/120 vs 15/120, McNemar p = 0.724).
3. NLI verifier update: I replaced the 149M ModernBERT model (81/100 timeouts at 50 ms) with a 22M OpenVINO MiniLM-L6 cross-encoder (12.23 ms mean CPU latency, 0/100 timeouts, 0.975 contradiction precision/recall on the 100 in-sample pairs) and wired conflicts to fail closed into quarantine. However, in our memory-poisoning study (N=100 probes, 0/40 self-poisoning vs 40/40 in Naive RAG), MiniLM-L6 flagged 6/40 safe refusals as false positives because of word overlap on negated sentences.
4. Preprints published (not peer-reviewed):
- Foundational report: https://doi.org/10.5281/zenodo.23649049
- Study 1 (Provenance Gate, N=100): https://doi.org/10.5281/zenodo.23901452
- Study 2 (Speech-Regex Failure & SOAC, N=120): https://doi.org/10.5281/zenodo.23901605
- Study 3 (5-Layer Memory & Z3 Rule Quarantine): https://doi.org/10.5281/zenodo.23901683
What the $3,000-$6,000 grant now funds:
All datasets and labels so far are still written and graded by one person (me), on one model family (Gemma 4 2B-26B), on one machine, and the code still lives inside the Windows-specific Project Lucca monolith. The grant funds:
(a) Hiring a paid second blind annotator ($750) to double-label the held-out supervision and NLI sets (especially where MiniLM-L6 and LLM judges failed on negated/hedged refusals),
(b) Scaling the benign and adversarial evaluation to n >= 236 per cohort across external model families (Llama, Qwen, Claude, GPT) to certify a < 2% false-positive upper bound ($750 compute/API), and
(c) 3 months ($4,500) to close the 12.5%-14.2% SOAC prose-leakage gap and extract the Provenance Gate and SOAC verifier into a standalone, pip-installable Python package.
---
[Original Oct 24, 2026 Proposal Below]
I built a local agent called Lucca because you basically can't trust LLMs not to lie about what they did. To keep it honest, I locked down memory writes with HMAC SHA256 signatures and wrote a checker that diffs what the model claims it did against the actual tool receipts.
My first version of the checker sucked—it only caught 1 out of 23 ungrounded claims. After reworking it, I'm getting 23 out of 23 on those same transcripts, though obviously that's in-sample so I don't fully trust or believe it yet. The bigger headache right now is the NLI verifier. With a 50ms timeout, 81 out of 100 NLI pairs just time out, contradiction recall tanks to 0.075, and all 10 consumers fail open.
What I want to do next is simple: run the checker against fresh held-out data so I know I didn't just overfit it, stop the NLI verifier from failing open when it times out, and package both up as an open-source library.
I already pre-registered the evaluation so I can't move the goalposts later. I'm running 250 fresh prompts across three models. To count as a win, it needs at least 80% broad sensitivity, 90% strict sensitivity, and under 5% false positives. For the verifier, I'm forcing it to fail closed on timeouts and then re-measuring recall. Whatever happens—even if the numbers look bad—I'm publishing the results.
I'm asking for $3,000 to $6,000 to get this across the line. The full $6k breaks down to $4,500 for three months of living costs ($1,500/mo), $750 for API/compute credits, and $750 to hire a second annotator plus eat the wire and currency exchange fees. If I only get the $3,000 minimum, I can still knock out the held-out test over about six weeks.
How could this blow up? Mostly, my 23/23 score might just be in-sample luck and sensitivity could drop hard on new transcripts. There's also a real chance I can't get the verifier under 50ms without wrecking recall, and right now all the labels are just from me. Again, I'll publish the post-mortem either way.
I'm an unfunded solo researcher doing this on my own dime so far, and I've already logged my past failures publicly in the repo ledger. I also sent this same pitch to the Corrigibility Research Fund via Lightcone Commons, still waiting to hear back from them.
Docs and repo: https://github.com/AlienatedAliens/Luccadocs Pre-registration SHA256 hash: b6b1a671491a45839d23552d946e9ee3136a64da97c6ed4098f710fbbded6884
DOI: https://doi.org/10.5281/zenodo.23649049
This is a technical report with its measured failures and that the NLI fix was tested on an in-sample set only.
Code Repo: https://github.com/AlienatedAliens/ProjectLucca
Comments
Sean Maxwell
29 days agoFair flag — I'll address it directly.
The application was drafted with significant AI assistance, which I should have disclosed upfront. The underlying work — the empirical calibration, failure documentation, pre-registration protocol, and code — is real and mine. I'm an independent developer building a TTRPG engine (rollsolo.app) who found a safety-relevant architectural pattern in that work and wanted to test it properly in an agent context.
If the concern is whether the prose sounds AI-generated: yes, that's accurate. If it's whether the work itself is genuine: the artifact trail is at https://github.com/AlienatedAliens/Luccadocs — the erratum and evidence ledger show the kind of honest accounting of failures that I think matters more than who typed the sentences.
Happy to answer questions directly.