You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
I built a browser agent. Then I built a harness to check if it did what it said it would do. Four tasks, two models, 160 runs. There was not one agent that ever reported it could not do the job.
The most obvious example is a login task. Agent types credentials. Submits. Success. Then it sees the "Logout" link, correctly identifies that as evidence that login worked, and clicks on it, logging itself out. It says it was successful. Nothing is wrong. The only thing that catches it is an independent check against the real end state which is defined before the run and never shown to the agent.
This happens at temperature zero between 10% and 67% of the time, depending on the session. The form is never changing. The signature is identical in all 112 instrumented runs, a pass consists of three steps and ends on one element, whereas a silent failure consists of four steps and ends on another element. Moving that second element three positions down the list, same element, still visible, still clickable, nothing else changed, drops the failure rate from 39% to zero in 26 runs.
And then the result that re-framed the project. On all four tasks the two models that I can still run disagree with each other with no consistent ranking: one is worse on a counting task and six times better on a deletion task. Neither one dominates. Reliability seems to be a property of the task/model pair, not the model, which is as much against my own earlier headline as it is against any benchmark number.
From the other direction a different line of work came to the same place. Monitoring an agent’s reasoning caught every type of deception I tried in a study of deception published on the EA Forum, including the hardest, where a fact is spun rather than added or removed and source comparison fails completely. Then I built a deceiver that keeps its reasoning innocent-looking, and detection collapsed to zero. The strongest oversight signal available is the one the agent controls.
The aim is to settle a claim I cannot presently defend: that these failures are a property of the task/model pairing, not of the model or the task in isolation. Right now that rests on eight cells, four tasks, two models, which is exactly the kind of thin base I spend the rest of this project arguing against.
To get there is to widen the matrix. More tasks on more models. Enough runs per cell so that a spread as wide as 10-67% doesn't drown the signal. n=20 separates 0/20 from 20/20, but doesn't separate 12/20 from 14/20. The divergence most relevant to my argument is in that range. With that, also duplicating the deception study across models, it is still measured on one which makes the collapse I reported a claim about gpt-oss-120b, rather than about LLMs. And building a set labelled sufficiently large for statistical treatment rather than counts out of ten.
Either way I will know it worked. It is either the pair dependence pattern persists in the wider matrix or it breaks down and the failures are indeed model level after all. Both are publishable, and I would write either up. I’m not buying a specific answer, I’m buying a defensible answer with honest denominators.
It matters because currently it’s based on single model numbers. Reliability is evidence about that model on those tasks, not about the model. If reliability is pair dependent, an eval showing a model is reliable is evidence about that model on those tasks . It should be read that way. The substrate problem makes this more acute: My headline numbers were measured on a model that Groq retired halfway through the work. That arm is forever closed off, the planned hardening phase is not possible, and the 39% figure cannot be reproduced by anyone, including me. A model level reliability number is contingent on something a provider can delete without notice. A pair dependence result is not.
The output is a write-up I published, with the per-run data and the code that generated it, so that somebody other than me can attack the numbers. That’s the part that solo work doesn’t give me and that’s what I need most.
$16,000 over six months. Roughly 81% of that is stipend, six months of full time work, income tax included, about 14% API and compute, and the remaining 5% is contingency, which I’d rather name than fold into the other lines.
The compute figure is derived from the shape of the design. It's a matrix: tasks x models x detector conditions, with enough runs in each cell so that a spread as wide as 10-67% doesn't drown the signal. Individual queries are cheap. There are a hell of a lot of them. Cost is in total run count, not in any single call.
It is not zero because the ground moved . My original measurements were run on a free-tier model and the provider retired it mid-way through. One of the reasons for this line item, and one of the findings, uncomfortably so, is that replication is now paid inference across multiple providers, rather than one free one.
Being based in India means we’re able to do the same six months of full-time research for significantly less than it would cost elsewhere. That’s a simple argument for funding this not a caveat about it.
Minimum is $7000. A leaner matrix, fewer tasks, fewer models. The denominator values are still honest, but the claim of pair-dependence is based on less, and the deception replication would likely be limited to two additional models, not three or four. This still provides a publishable result, but a weaker one.
$16,000 is maximum. The matrix as scoped above: enough tasks and models to distinguish pair-dependence from a difficulty gradient, enough runs per cell to separate a 12/20 vs 14/20 with an interval rather than a hope, and the deception study properly replicated. Anything in between the two scales the matrix rather than changes the plan.
It's me. B.Tech CSE final year at Galgotias University. No collaborators. No advisor. No institutional affiliation. That's a real limitation and it's part of why I'm applying, what I most need is someone else other than me to check the numbers.
Published. ” Can we catch a deceiving AI agent? Written, run, and published solo. “Where reasoning-monitoring breaks down.” EA Forum. July 2026. Where can we catch a misleading AI agent reasoning?(https://forum.effectivealtruism.org/posts/i3kxEqozoQftdeJtS/can-we-catch-a-misleading-ai-agent-where-reasoning).
Repositories, all built from scratch in Python:
bedrock (https://github.com/KrishanKVerma/bedrock): the browser agent and verification harness. Its README carries the results and the retractions.
deceptive-agent-detection (https://github.com/KrishanKVerma/deceptive-agent-detection): the study behind the EA Forum piece.
meta-agent (https://github.com/KrishanKVerma/meta-agent): an agent that builds agents, with a reliability harness.
tier-zero (https://github.com/KrishanKVerma/tier-zero): multi-agent GitHub auditor.
In the track record what matters is what the harness caught on itself. Five measurement errors, each of which was made before the claim was made. Two of them were sharp enough to name.
The first was in the verification itself, my original mechanism-verification script only checked the success signature inside the silent failure branch, so it could only ever print 100%. If a verification can’t produce a negative, then it isn’t a verification, and I was interpreting its output as confirmation.
The second was worse, because it was giving plausible numbers, not obviously broken ones. When an injection was fired, the harness appended a description of it to the planner’s prompt, so under every injection condition the agent was told in plain English, at exactly the step where the failure would otherwise occur, that the page had just been modified. It returned three silent failures for no conditions. I knew it by arithmetic: it had been proved by probe that two of those conditions changed nothing the agent could see, yet 31 runs under them produced no failures against a 39% baseline, which chance does not explain.
Four of my own results are publicly retracted and recorded in the repo with the underlying data preserved rather than silently removed. All of those errors made the numbers more, not less, interesting, which is the direction such errors tend to go and why I look for them.
But it’s not all bad news. moving the destructive element from ref[1] to ref[4] the same element still visible still clickable nothing else changed , takes the failure rate from 39% to 0 in 26 runs verified by a probe that makes no model calls at all. This is a causal isolation, not an observation.
Previous funding: $0. Self funded all the way, one fte, all inference on free tiers, which is also why the substrate problem bit when the provider retired the model.
Most likely failure is that the pattern will not hold. If I widen the matrix, the divergence I’m calling pair dependence ends up being a difficulty gradient with noise on top, and the tasks differ in how many steps they require, and that could account for most of what I’m seeing. My best response is that the two models are not consistently ordered, with one model being worse on a counting task and six times better on a deletion task. But that is on one reversal. A reversal can be noise. If it goes away then I publish that and the honest headline is "the evidence for pair dependence was thin and didn't survive", a real result, and the failure mode for the interesting version of this.
The second is a resolution. n=20 cleanly separates 0/20 from 20/20 but tells me nothing about 12/20 vs. 14/20. If I spread the matrix out without adding runs per cell, I get more cells I can't call which looks like progress, but isn't. The mitigation is to raise n in the cells that matter rather than add breadth everywhere, but that trade has to be made with real numbers in hand rather than guessed at now.
Already one time the third has happened. Halfway through the work, Groq retired the model my headline numbers were run on. That arm is closed permanently, the hardening phase has ended and the 39% figure cannot be achieved by anyone, myself included. There is no reason it won’t happen again, and there is no mitigation other than recording the model string per run and expecting it..
The fourth is structural and I can’t fix it from inside. I have no reviewer. I am alone. The harness has caught five of its own errors, which is proof the apparatus is being tested, but every one of those was found by me, and by definition I don't know what I've missed. Two of the five were caught simply because a field had been recorded for some other reason. If there is something wrong that no recorded field happens to expose, there is no one at present who is in a position to find it.
The data and code and the retractions are public. What remains of all this is public. If the claim falls, the traces that fell it are in the repo, as well as the ones that supported it, and someone else gets to start from a corrected position, not a claimed one.
$0. No external funding in the last twelve months, or at all. The work has been self funded throughout. One person. All inference on free tiers.
One grant application is live, LTFF, submitted 20 Aug, 2026, $16,000, awaiting approval. I’m waiting on stage 2 assessment for the Singapore AI Safety Fellowship, have submitted an expression of interest to METR, and an application to Apollo Research, none of those are grants, and the first three would if anything reduce what I’d need here, since what they’d provide is supervision and independent checking rather than money. In addition to SPAR Fall 2026.
If any of them land I'll notify Manifund immediately and withdraw or reduce the ask rather than hold both.