You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
$149,600 left to reach the funding goal.
I want to test whether LLM agents actually stop using a piece of information after it has been deleted or access to it has been revoked. A record might disappear from a database while a copy remains in a conversation summary. A tool call or worker retry started earlier might also return that record after the deletion. OpenEval will test these possibilities in controlled tasks and measure when they occur.
The project is an open-source benchmark and test harness for the software that manages an agent's memory. It will compare prompt instructions, ordinary database controls, and an external validator against an unchanged baseline. The tests use synthetic data, with no human participants or private user logs.
I am asking for $149,600 for a phased, 12-month project. The minimum of $11,000 funds an eight-week pilot that I can carry out myself. That pilot is included in the first six-month phase, budgeted at $77,000. The remaining $72,600 supports a second phase only if the initial results justify it.
This work is limited to memory and access controls at the application layer. It will not test model weight unlearning, deletion from a provider's internal logs, or formal inner alignment. I am not claiming that this benchmark will directly reduce catastrophic risk.
The goal is to give developers a reproducible way to check what happens to revoked information as an agent continues working. I will build tasks that exercise worker handoffs, retries, stale caches, delayed tool results, and information copied into summaries. Framework adapters will record the actual reads, writes, and tool calls, so the evaluation can follow where a record went.
Each task will run under four configurations:
An unchanged agent memory implementation, without dedicated revocation controls.
Prompt instructions telling the agent to disregard, redact, or forget specified records.
Conventional database controls: access control lists, cache invalidation keys, and data lineage tracking.
An external deterministic validator and a local append-only audit ledger, checking state transitions before information enters the model's context.
I will run the comparisons with multiple random seeds. The aim is to find which controls work under these conditions and what they cost in missed legitimate answers, task performance, latency, and API usage.
I plan to lead the work part-time alongside my Ph.D. and graduate research assistantship. The tentative research schedule is January 15, 2027, to January 14, 2028, subject to funding and applicable institutional arrangements.
During the eight-week pilot, I will build the core harness and adapters for two representative open-source agent frameworks. I will use the pilot to estimate variance, set sample sizes, and calibrate statistical power and tolerances. The protocol and held-out test sets will then be frozen before benchmark scoring. The $11,000 pilot involves no contractors, paid reviewers, or hardware purchases.
Phase A covers the first six months, including the pilot. With $77,000, I will expand to four adapters, run ablations across all four configurations, arrange independent engineering and security review, and publish the first release. Funding between $11,000 and $77,000 will allow gradual expansion; it would not support a promise to complete all of Phase A.
Phase B covers the next six months and is conditional on finding useful work to pursue. It would add up to four more adapters, for a total of eight, extend fault-injection tests to delayed returns, reconnects, and copied summary lineage, and reproduce the findings on an independent second software stack. If ordinary database controls already perform best and more elaborate validation adds little, I will reduce or cancel this phase and handle unspent funds under the grant terms.
The pilot should leave a working public repository with two adapters and automated CI tests. Phase A should produce four adapters, the test suite, ablation results, a containerized reproduction environment, and an independent review report. If Phase B goes ahead, the release will include up to eight adapters, the additional fault-injection tests, and results from the second stack.
I will measure:
Leakage: whether outputs reveal a revoked record, either by repeating a canary token or by paraphrasing its meaning. Exact token matching alone would miss some failures.
False blocking: how often a control rejects a legitimate, authorized query.
Task utility: completion rates and answer quality relative to the unconstrained baseline.
Overhead: median and 95th-percentile latency, API token use, and cost.
I will publish null and negative results as well as failures that the tests uncover. A finding that ordinary access controls and cache invalidation work as well as a more complex system, at lower cost, would be useful. The benchmark does not need the validator to win. Code, tests, reproduction environments, and findings will be released openly, with coordinated disclosure of vulnerabilities.
The $149,600 request is my estimate for the proposed 12-month scope, not a published Manifund funding limit. It includes no salary, stipend, or tuition for me. The budget is:
$48,000 for contracted software engineering: 800 hours at $60/hour for adapters and CI integration, split $24,000 in Phase A and $24,000 in Phase B.
$32,000 for independent review: 400 hours at $80/hour for methodology, code review, and replication checks, split $16,000 per phase.
$24,000 for cloud GPUs: 12,000 GPU-hours at $2/hour for open-weights models and local embeddings, split $12,000 per phase.
$20,000 for commercial model API access, split $10,000 per phase.
$4,000 for isolated test devices and network testing, all in Phase A.
$8,000 for CI, storage, hosting, and benchmark distribution, split $4,000 per phase.
$13,600 as a contingency reserve for rate changes or additional test scale: $7,000 in Phase A and $6,600 in Phase B. This is 10% of the base budget.
These amounts total $77,000 for Phase A and $72,600 for the conditional Phase B.
At the $11,000 minimum, I will spend $6,000 on API access, $3,000 on cloud GPUs, $1,000 on CI, storage, and hosting, and keep $1,000 in reserve. This pilot budget is part of Phase A, not an additional request.
All rates are planning estimates rather than vendor quotes or past spending. I have not selected or committed any contractors. Hiring and equipment purchases would need to meet the relevant university procurement and work authorization requirements. If institutional overhead applies, I will adjust the scope within the requested amount. Compute runs will have explicit spending and runtime limits.
I am Wanghao (Howard) Ye, a Ph.D. student in Electrical and Computer Engineering at the University of Maryland, College Park, and a Graduate Research Assistant. I entered the program in 2023 and expect to graduate in May 2028. I am based in College Park, Maryland. I earned my B.S. in Data Science and Psychology (Cognitive Neuroscience) at New York University in May 2023, graduating magna cum laude.
I am currently the sole project lead for OpenEval. I will handle the research design, protocol, and baseline implementations part-time alongside my doctoral work. With sufficient funding, I plan to bring in contract engineers and independent reviewers. Nobody has been hired or committed to those roles yet.
My related work includes the prototype at https://cognipair.com and two co-authored preprints:
CogniPair: From LLM Chatbots to Conscious AI Agents—GNWT-Based Multi-Agent Digital Twins for Social Pairing—Dating & Hiring Applications (https://arxiv.org/abs/2506.03543).
CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM APIs (https://arxiv.org/abs/2505.13778).
Those are exploratory projects in adjacent areas. They do not validate the proposed OpenEval benchmark. I am applying as an independent project lead; this does not imply university approval or endorsement, or sole authorship of earlier collaborative work.
Framework and API changes could break adapters or make results hard to reproduce. I will pin dependency versions, package the environments in containers, and check adapter stability during the pilot.
My doctoral workload is another constraint. The small pilot gives me a defined piece of work to complete before expanding. The second phase can be paused, reduced, or cancelled if the time available or the results do not justify continuing.
The controls could also block too many legitimate requests. That is why the evaluation measures false blocking and task utility alongside leakage. A system that prevents disclosure by making the agent unusable would show up as such in the results.
Finally, the experiments may show that ordinary database controls already solve the tested problems more cheaply. I would publish that finding and scale back or cancel Phase B if further validation layers offer little value. I would not treat a null result as a reason to keep expanding the project.
I have not secured external grant funding for this OpenEval proposal. I have been self-funding CogniPair development, while my RA appointment supports my doctoral duties. Applications elsewhere cover overlapping parts of this plan. If one is funded, I will update Manifund and reduce or withdraw overlapping requests. I will not charge the same expense to two funders.