We will test whether circuits in protein language models can detect function-preserving redesigns of known toxins that evade homology-based DNA-synthesis screen
What this project is
DNA synthesis is the point at which an AI-designed protein sequence transforms from a digital file into a biological molecule. Producing the protein generally requires the corresponding synthetic DNA. Providers screen orders against databases of known toxins and pathogens, making this a central biosecurity safeguard.
Current screening relies mainly on sequence similarity. Ordered DNA is translated into protein sequences and compared with known proteins of concern using sequence alignment, profile hidden Markov models, and short-sequence signatures. These methods work well for close matches but are less reliable when protein design tools alter sequence while preserving structure or function. Recent work has shown that redesigned proteins can diverge substantially from known sequences and evade standard homology-based screening [1].
This means advanced AI could pose new risks. The very feature that makes AI-driven protein design so powerful—changing a sequence but keeping the same function—can also make it harder for current screening systems to spot dangerous proteins. Even if we grow our databases, the core issue remains: a protein can look different yet do the same thing.
This project asks: can we screen for what a protein actually does, not just what it looks like? Even when a protein is redesigned to keep its function, it still relies on certain key features, like catalytic residues, binding sites, and structural contacts, to work. These features might still show up in screening, even when the overall sequence looks unfamiliar.
Our prior work provides a direct way to test this idea. ProtoMech uses cross-layer transcoders to identify circuits inside the protein language model ESM2 [2]. We found that these circuits respond to catalytic motifs, binding pockets, and regions involved in structural stability. We will test whether these signals persist across redesigned versions of known toxins and identify variants that homology-based methods miss.
Our main idea is that these circuit-level signals could still be picked up by screening, even after traditional similarity checks stop working well.
The grant will support a six-month computational study using publicly available sequence, structure, and functional data on known toxins.
[1] Bruce Wittmann, Tessa Alexanian, Craig Bartling, Jacob Beal, Adam Clore, James Diggans, Kevin Flyangolts et al. "Strengthening nucleic acid biosecurity screening against generative protein design tools." Science 390, no. 6768 (2025): 82-87.
[2] Darin Tsui, Kunal Talreja, Daniel Saeedi, and Amirali Aghazadeh. "Protein Circuit Tracing via Cross-layer Transcoders." International Conference on Machine Learning Research (ICML) 2026.
<https://protmech.github.io>
<https://github.com/amirgroup-codes/ProtoMech>
Theory of impact
Advanced AI could make it much easier and cheaper to design biologically active proteins. If AI-powered redesign outpaces the screening systems used by DNA-synthesis providers, it may get harder to spot and stop harmful proteins before they are made.
This project looks at one possible solution. A circuit-based screen wouldn’t replace current sequence checks, but could add another layer of safety for proteins that don’t look risky but might act in dangerous ways. This method could be used as a backup for cases where the risk is unclear, helping keep costs down and preventing false alarms from getting in the way of normal work.
If this approach works, it would show that mechanistic interpretability can directly help screen for biological risks—not just help us understand how models work. The method could then be built into open-source screening tools or tested by synthesis providers alongside current sequence and structure checks.
Even if this approach doesn’t work out, that is still valuable to know. If circuit signatures fade away just as quickly as sequence similarity or don’t add anything new, we will know not to rely on this method. That would help avoid wasting time and resources on an ineffective technique and make it clear where the limits are for using protein model interpretability in biosecurity.
How the money will be spent
$30k: This would cover a smaller six-month study focused on building benchmarks, analyzing circuits, and comparing results with homology-based screening. However, it would include fewer protein families, redesign methods, and robustness tests.
$50k: This amount would fund the entire project. It would allow for broader toxin coverage, more redesign methods, and comparisons with various baseline approaches (such as sequence, embedding, and structure). It also covers ablation studies, GPU costs, and the public release of the benchmark, prototype, and technical report.
Comments
Amirali Aghazadeh
3 months ago@grantmaking-ai @manifund I wanted to follow up again regarding the agreement for our funded project. We are ready to begin the work, but we cannot initiate the project until the agreement is executed and the funding is released to Georgia Tech Research Corporation.
Could you please let us know whether the agreement can be provided as a PDF for the required Georgia Tech signatures, or whether there is another way to move this process forward? We would appreciate resolving this as soon as possible so that we can start the project.
Best,
Amirali Aghazadeh
Caroline Ellison
3 months ago@amiralia Hi! Sorry for the delayed response--we just rolled out a feature that allows grant agreements to be signed externally by organizations. I've reset your agreement so you can redo it as an organization: on https://manifund.org/projects/safeguarding-dna-synthesis-from-biological-risks-posed-by-advanced-ai/agreement, enter Georgia Tech Research Corporation's details and the email of GT's authorized signatory, and they'll receive a link to review and execute the agreement (they'll get a copy for their records). Once that's signed, you can withdraw the funds to GTRC at https://manifund.org/withdraw.
Let me know if you need the agreement as a PDF instead, or if you run into any issues!
Amirali Aghazadeh
3 months ago@carol @grantmaking-ai @manifund Thanks for this. Georgia Tech Research Corporation (GTRC) has signed the agreement; we are good in terms of sign-off. GTRC, however, is not able to withdraw the money using the link above. Is it possible to obtain payment via ACH?
Caroline Ellison
3 months ago@amiralia yes! you or they can fill out this form: https://airtable.com/appOfJtzt8yUTBFcD/shrI3XFPivduhbnGa it asks for ACH details (bank account name, address, routing number, account number) as well as Manifund username and project link.
If that doesn't work feel free to just email the bank account info (hi@manifund.org) and we can handle it.
Amirali Aghazadeh
3 months ago@carol Many thanks. GTRC has confirmed that they have submitted the ACH information via the Manifund website. I guess we need to wait for a couple of days now before the payment lands. Thanks much. I will let you know if we need any additional help.
Caroline Ellison
3 months ago@amiralia Yep I see the ACH info, thanks! We'll send the payment today and then you should receive it within a few days.
Amirali Aghazadeh
2 months ago@carol Can you please confirm if the payment has been sent so that I can track it down here at Georgia Tech. Many thanks.
Caroline Ellison
2 months ago@amiralia Hi, yes this was sent by ACH on November 2.
grantmaking.ai
4 months agoOffering this grant per recommendations by Ryan and Marcus:
https://app.grantmaking.ai/projects/b89d33b5-560f-4537-b9f2-997f35116fde