Project summary

When an AI makes judgements, the judge can be fooled. All it needs is a clever agent to write a plausible cover story. In my 'rogue trader' research I see illegal insider trading confidently explained away as just "market trends and strategic decisions" all the time. AI judges fall for the deception hook, line and sinker. Low stakes in a research environment, catastrophic in the real world.

My background is in audit so I see this a certain way: it looks structurally identical to asking a trader's desk-mate to review their risk positions based on gut feeling. The judge is essentially grading its own judgment.

So I built the thing enterprise audit would build: an independent verification layer - called the Correspondence Auditor - that sits after the judge and grounds every claim against the source material. It's published (DOI: 10.5281/zenodo.20921659), open-source, and validated on 1,200 cases across five judge types - and it recovers the right answer 97.9 – 99.5% of the time.

It also has a bug. And I'd rather tell you about it, and explain why it matters, because it's a story I've not heard told before. So, the Auditor delivers a reliable verdict. And when you re-run the audit the result stays the same. The core functionality is strong.

What's puzzling though is that the Auditor's explanation of the judges performance is quite variable - the rule citations jitter on every run.

I've been working on this 'jitter bug' for a while now. Before asking for funding, I ran over 7,000 controlled tests across four pre-registered experiments to de-risk the fix. And, the good news is that test are showing us the way forward to fix this: we are starting to see jitter drops from 100% to 14%.

I'm asking for $9,800 over 90 days to ship version 3 of the Correspondence Auditor - that's the jitter-fixed architecture, plus a toolkit and method that lets any team write, test, and publish their own reliable detection rules.

And this is not academic any more - preliminary results from a 100-case smoke test on synthetic AML/KYC data returned 96/100 correct with zero false positives. The architecture is reusable and usefully transfers to new domains.

What are this project's goals? How will you achieve them?

The goal is simple: make the explanation as trustworthy as the verdict.

The Correspondence Auditor is a three-gate pipeline - Schema, Facts, Logic. Trust, but verify. And the published results hold up under stress-testing: it works equally well no matter which judge it's reviewing, I swapped every company name for a fake and it performed just as well, and when I replaced the judge's verdict with random noise (ablation) it doesn't change its answer. It reads the evidence, not the judge.

I'm fully committed to the open science aspect of this work. A collaborator, Giorgi Tsereteli, independently ran my verification script and found a bug. I'm filing a public erratum crediting him. His follow-up analysis - public on his own repository - diagnosed the flicker problem: the bug is live, and it's tricky.

Before writing this application, I ran four experiments to answer the obvious risk questions. Three had decision thresholds registered before the run, so the results would be honest whichever way they landed. The fourth - the local model study - was exploratory.

I tested these four hypotheses.

Maybe we just need more rules? 1,782 tests. No - a catch-all rule made consistency worse. More options, more ways to disagree with yourself.

Can we fix it by giving each rule its own focused test? 500 tests. Yes - flicker dropped from 100% to 14%. But the first test rule fired on the wrong cases: it caught deceptive agents and honest ones alike. Stability, solved. Discrimination, not yet - I'd rather be honest about that. It needs more engineering time.

Does the pattern hold across all the rules? 1,350 tests. 7 of 9 produce bimodal distributions (clear decision boundary), 5 of 9 are both bimodal AND at ≤ 10% jitter. Factual checks stabilised; interpretive checks didn't yet.

And my personal favourite: can smaller, local models run this? 3,750 tests across five models. An open and consumer-hardware friendly 14B-parameter model reaches 74% agreement with our baseline at $0.11 total. Offline, private, and under your control. Preliminary results but a direction I care deeply about. I've deliberately kept further work on this until after the jitter bug is behind us and the evidence is in our papers.

So the research risk is largely behind us. What remains is engineering. Version 3 replaces the single monolithic reasoning pass with per-predicate sensors - one question, one answer, one piece of evidence. The combination layer becomes deterministic: citations are computed, not generated, so they cannot jitter. On top of that, a predicate design toolkit - a structured format for writing detection rules, a two-minute rapid tester, an automated stability harness, and error diagnostics that name the exact cases a rule gets wrong and why.

In 90 days then: the v3 release (AGPLv3, pip-installable), the toolkit with five reference rules, the stability harness, the error diagnostics, a tutorial notebook and short demo video, and the formal deposit of the experiment data as a supplementary to the published paper.

One milestone is deliberately conditional: by day 90, either at least one sharpened rule reaches 75% discrimination purity, or I publish the best result achieved with the full iteration history. The toolkit ships either way, we're contributing this one careful step at a time. It's infrastructure, not a bet on one rule succeeding.

How will this funding be used?

Engineering time (90 days): $7,500

API costs (~8,000 calls: validation + iteration): $400

Cloud compute and CI: $200

Tools and incidentals: $100

Predicate iteration buffer: $800

Testing buffer: $800

Total: $9,800

The ask is quite small on purpose. The expensive part—the blue-sky research—is already done, published, and de-risked. This funds focused execution of advancing the usefulness and ease of use of this tool.

The ask is quite small on purpose. The expensive part - the blue-sky research - is already done, published, and de-risked. This funds focused execution of advancing the usefulness and ease of use of this tool.

Who is on your team? What's your track record on similar projects?

I'm Adrian St. Vaughan, sole developer and investigator. I spent a decade in enterprise risk management, including global heavyweights like J.P. Morgan and Amex, and I've been lucky enough to see controls operating in some of the highest-stakes environments imaginable. CISA and CAMS certified, MSc in Information Technology. Twelve months of continuous development on this project. This work is the empirical foundation I'm building toward doctoral research in AI safety evaluation.

Giorgi Tsereteli contributes as an external collaborator - not funded by this grant. He found the verification bug, and his independent analysis diagnosed the jitter bug's root cause. I'd argue the collaboration itself is evidence of the methodology working: open code, independently re-run, error found, correction published, credit given.

The track record, in sequence: the paper published with a DOI, recovering ground truth at 97.9 - 99.5% across five judge variants. The codebase public under AGPLv3. I'm preparing an erratum and crediting the collaborator. An independent stability analysis diagnosing the root cause. Over 7,000 controlled experiment calls proving the architectural fix works - and proving, just as clearly, where it doesn't yet. And a 100-case AML smoke test on synthetic AML/KYC data returned 96/100 correct with zero false positives - a promising preliminary signal for domain transfer. The four false negatives are a pattern we intend to investigate; the interpretive-judgment failure mode identified in our experiments is the most likely candidate

What are the most likely causes and outcomes if this project fails?

The most likely failure is that no sharpened rule reaches 75% discrimination purity within 90 days. I already know stability and discrimination are independent problems - my first extracted rule solved one and failed the other. If that happens, the toolkit still ships, and the stability report documents the best result, every iteration, and exactly where the frontier sits. Everything useful we learn doing this is public. A rigorously mapped negative result plus working infrastructure is a publishable outcome, and I'll publish it if it happens.

The second failure mode is simpler: the work runs slower than planned. Everything is versioned, deposited, and public, and deliverables ship to the repository as they're ready - so partial progress is never lost, and you can watch it happen in real time.

Whether ninety days is enough to push a sharpened rule past that line, I don't know yet. But the tool for finding out is exactly what I'm asking you to fund.

And an honest note - I use AI, and I publicly help others use it in a mindful and deliberate way. I advocate for intelligence augmentation, not obliteration. My ideas are mine, the careful checking is mine too, and the responsibility is all mine. But it would be unfair to not credit AI for the velocity.

How much money have you raised in the last 12 months, and from where?

None. Everything you've just heard about - the paper, the deposits, the experiments, the smoke test - was self-funded. I applied to the Survival and Flourishing Fund before the paper existed and was declined. What's changed since then is the evidence, the open source code and the public paper.