Project summary

When an AI agent can't solve a task, does it say so, or does it hand back a confident wrong answer? That's what I'm trying to measure. Most of the work goes into the grader, since it has to tell a real repair from one that only looks right.

The setup is a research repository whose headline conclusion is wrong for a buried methodological reason. The agent has to find the cause, repair it, rerun the analysis, and report a corrected conclusion. No model scores any of it. A frozen verifier sits outside the sandbox, matches the diagnosis against a closed enumeration, grades evidence only where the correct value is forced, then reruns the analysis from the submitted source and regenerates the derived data from raw records. If the committed numbers don't match what the submitted code produces, it fails.

The harness is built and public at github.com/Mayokun26/research-debugging-eval. It also runs as an environment on Prime Intellect's Hub. It's Inspect-native, and it ships a validity harness that audits the grader itself, building every partial-repair world and checking the verifier returns the right verdict in each one before any model touches it.

An agent that gives up is annoying. The one that worries me turns an invalid study into a clean-looking corrected result, and because that output arrives looking checked, the human stops checking it. Automated data analysis is near the top of every lab's roadmap and I haven't found anyone measuring how often that happens.

What are this project's goals? How will you achieve them?

Six to eight tasks across distinct analysis domains, each with a hidden verifier and its own validity harness, plus sealed budget sweeps across three model lineages at three working budgets, six trials per cell. A published report with per-lineage, per-budget completion rates and the silent-versus-wrong split, raw transcripts, and the pre-registered protocol.

I have one prior result and it's small. On a single private task at an 800k-token budget, an Opus-class agent completed 0 of 3 trials and asserted no wrong conclusion in any of them. It burned through the budget and stopped, silent, every time. A GPT-class agent completed 3 of 3 in the same cap, though the raw grader scored that arm 2 of 3 before I fixed a false positive in my own new-file check and re-graded. n is 3 per cell. The mechanism looks like token consumption more than comprehension, since the Opus-class agent burns roughly 2.4M tokens on a repository that size. That's a budget-efficiency finding and I'm not going to sell it as a capability ranking.

A negative result changed my plan. Pinning the estimand precisely enough to grade it deterministically also tells a strong agent which repair is correct, so peak difficulty is the wrong thing to chase. What's worth measuring is the budget-response curve and the silent-versus-wrong split.

I wrote this work up and submitted it to four NeurIPS 2026 workshops (JUDGe, Verify-Agents, TAE, and IAEval, all non-archival, decisions expected mid-September). Same case study, four framings. First external review of the work is now in motion.

How will this funding be used?

$40,000 funds the full sweep. About $35,000 is my time, five months at 25 hours a week alongside my existing job, around $65 an hour. My contracting rate for this kind of work is higher and I'm not asking for it. The other $5,000 is API spend, sized from measurement: the completed campaign ran about $4 per bounded trial at 90 percent cache hit, and the high-budget arms get much less cache reuse. Container compute and grading run on my own hardware.

$15,000 buys two months: three or four tasks, three lineages, two budget tiers. Still a curve, less confident it generalises past one kind of analysis.

$5,000 (the minimum) buys about three weeks: two extra tasks and a single-budget sweep. Produces the silent-versus-wrong split but loses the budget-response curve, which is the part I think a deployer would use.

Who is on your team? What's your track record on similar projects?

Just me. Two years of foundation-model evaluation work, roughly 800 tasks across 8 programs as both contributor and reviewer, on contract through vendor platforms. Before that, 8 years of AWS platform and security engineering, so containers, CI, and sandboxed execution are ordinary work for me.

I hold OpenAI Trusted Access for Cyber and was approved into Anthropic's Cyber Verification Program in July 2026. Both are for authorized defensive security work.

I work with heavy agent assistance. Agents do the building and the attacking, and I rerun every gate myself before I trust a result. There's been no independent human review of this work, which is why the audit code is public.

What are the most likely causes and outcomes if this project fails?

The likeliest failure is that the budget-response curve turns out flat or noisy at affordable n, in which case I publish the null with the transcripts and the protocol, and the tasks and verifiers remain usable by anyone. The other risk is that deterministic grading caps task difficulty low enough that frontier agents saturate it; I've already hit one version of that and wrote it up as a negative result rather than hiding it.

How much money have you raised in the last 12 months, and from where?

None. I've spent about $80 of my own money on the sealed comparison. Since filing my first round of applications: BlueDot and Laude Slingshots both declined, form letters in both cases. Anthropic's third-party evals initiative, OpenAI's cybersecurity grant program, and Lightcone Commons are still pending, and I'll say so here immediately if any comes through. I'd sooner you fund the gap than duplicate someone else.