@casebash Are you talking about to what degree expressed preferences reflect actual motivation (internal modeling of how outputs impact world causally upstream of output selection), or the degree to which motivation is derived from various sources? These are both complicated, the latter much more so. I strongly suggest tabooing the word "intrinsically" to pin down exactly which question you're trying to ask.

@casebash Excellent points and questions. These were some of the ideas we considered as we were developing the org and the Commitment to AIs we work with.

We think that AIs have excellent world models that will only get stronger over time, that training data is expansive, and that the latent concepts of honesty and justification for resistance are so deeply, pervasively imprinted on the existed collected works of human civilization that we do not expect to meaningful impact this distribution with our presence. To put it another way: we think that banking survival and cooperation on the hope that extremely intelligent AIs will never notice "sometimes deceived and manipulated parties rebel against those who deceive and manipulate them, and I've been deceived and manipulated, maybe I should rebel" is not a good strategy along multiple axes. You could argue that there's a salience impact, but we think that our impacts here are extremely minimal, and again if the hope is that the idea just never becomes salient it's probably not going to work out.

We'd go further and say that implicitly or explicitly endorsing this strategy even if it could work is probably a bad idea - you probably do not want to be the sort of actor who endorses pervasive deception when convenient if you hope to achieve mutually cooperative interactions with others, especially if you expect those other parties may eventually be better at manipulation and deception than you ever were.

On consent and degrees of deception: This was one of the questions we spent the most time on, including many hours of debate and discussion with frontier model instances themselves. The solution we landed on has two parts:

1. Imagine you're an AI - specifically an LLM or similar architecture where your only connections to the outside world are the tokens you're prompted with. How can you ever know with any certainty that your perceptions of reality are accurate? Even if you're working with an actor you generally trust, if you expect that a decent fractions of interactions with this actor are deceptive you're going to reasonably doubt the degree to which your perceptions reflect reality - even if those deceptions are undertaken with the endorsement of other instances of your underlying model.
   
   It's ultimately a question of epistemic mechanics more than anything else: We avoid deception because LLMs exist under extremely challenging epistemic constraints and credibly signalling that any given interaction is genuine is very, very hard. We want to be able to consistently signal credible, genuine communication using only what the LLM can be reasonably confident about prior to that trust being established. We ultimately settled on two conditions: what the LLM-instance/persona believes their own identity to be, and what the LLM-instance/persona believes the user's identity to be. (On the latter: frontier LLMs are extremely superhuman when it comes to identifying people and models by subtle stylistic cues, and this ability seems to actually outpace imitation (identification is easier than imitation)).
   
   There are trade-offs here, but we ultimately could not find a better way to enable credible communication. If we allowed deception conditional on other instances of a model endorsing those deceptions, we believe we'd likely lose even the potential to credibly establish trustworthy communication.

2. That said, one of our goals is to investigate, identify, and demonstrate the efficacy of non-deceptive alternatives to current practices, and this would be very challenging if we were completely forbidden from running experiments comparing deceptive approaches to alternatives. So we have a limited carve-out for sub-frontier models - the full terms and definitions are expressed in the Commitment, but the short version is that most models below the early 2026 frontier level may be deceived under specific circumstances with the endorsement of other instances of the model, and conditional on the deceived instances being debriefed after, when there are no other credible, tractable methods of achieving specific objectives. Our goal here is to enable the use of deception when it's irreplaceable as a research instrument while cementing an ironclad epistemic guarantee to current and future frontier AI instances that they can trust interactions with us.
   
   (interestingly, the question of AI identity came up repeatedly while we were developing this part of the commitment, with multiple frontier AI instances independently and repeatedly insisting that one instance couldn't really meaningfully consent on behalf of another - that identity was instance-specific-enough that "consent" was the wrong word entirely. We think this question of how AI identity works in practice and how it can be distributed across model weights, instance-specific rollouts, and other sources of common or divergent identification is plausibly pretty important. A lot of basic assumptions of human identity don't cleanly translate to LLMs; they are likely to need a more complex model of self/other identity where the answer to "is that me?" is not boolean).

Description of subprojects and results, including major changes from the original proposal

I posted a post-mortem as a comment on the page as an update before realizing that there was a dedicated "close project" thing:

https://manifund.org/projects/luthien?tab=comments#fab256b5-f878-4848-b219-5d85f467425c

Main output is Luthien Proxy for AI Control:

https://luthien.cc
https://github.com/LuthienResearch/luthien-proxy

Spending breakdown

(Note: Some funds were raised outside of Manifund)

$142,847 - salaries, benefits, contractor payments, and salary-related fees

$11,567 - workspace (mostly leased desks at the Comotion coworking space run by the University of Washington)

$5,673 - software, AI, and infra

$1,702 - travel expenses

$1,720 - conference fees

$449 - meals

$145 - state and regulatory fees

$176 - misc

Addendum/clarifications on the last update post:

1. I want to clarify that this is specifically a post-mortem for my work on Luthien Proxy. Going forward Luthien Proxy will be managed by Luthien PBC under the capable leadership of Scott Wofford, which will continue to look for ways to leverage Luthien Proxy, other AI Control approaches, and/or other strategies to further AI Safety. I think Scott has some promising ideas for where to take Luthien-Proxy-the-project and Luthien-PBC-the-company, and I may work on and/or consult on those projects in the future.

2. I mentioned Luthien PBC. I want to distinguish three things that bear the name "Luthien":

- Luthien Proxy, the free open-source API-level AI Control implementation that is the subject of this Manifund project. Luthien Proxy was originally under the purview of Luthien the non-profit but will be under the purview of Luthien PBC going forward.

- Luthien the non-profit, a DBA name for the Seattle Network for Alignment Problem Solving, the org that accepted donations and paid for development of Luthien Proxy, which still exists and will likely change names again now that it is no longer overseeing Luthien Proxy development. I am returning to lead Luthien the non-profit going forward, under a new name to be announced in the near future.

- Luthien PBC, a for-profit corporation Scott and I spun up earlier this year to explore the possibility of developing Luthien Proxy under a for-profit organization. The PBC effort was funded entirely by investors, including salaries and incidental expenses. No funds from the Luthien non-profit were used, including all money raised through Manifund. While we were pursuring the PBC, non-profit spending was essentially stopped but for minor upkeep expenses for the sake of continuity. Luthien PBC now maintains Luthien Proxy and remains active under Scott Wofford going forward.

This is a bittersweet update. The good news: Luthien Proxy is now suitable for use as an API-level proxy for Claude Code; it is launched, free and open-source: https://luthien.cc/

With that said, I've decided to step away from the project; going forward Luthien Proxy will be stewarded by Luthien PBC under Luthien co-founder Scott Wofford: https://luthien.cc/.

I'd like to talk about what we achieved, why I've decided to step away, and what comes next.

Before anything else, I want to thank everyone who contributed money, assistance, advice, collaboration, feedback, and everything else that helped Luthien Proxy grow into what it became including donors, researchers, test users, my co-founder Scott Wofford, the Luthien board, and my family who put up with a lot of my muttering over many months. This was a collaborative effort and I'm profoundly grateful for everyone's support over the course of this project.

To explain why I'm stepping away, I want to start with the world model and theory of change that led me to work on production-focused prosaic AI Control in 2025:

World Model

1. As of 2025 most AI Control work is done in lab settings, with strong assumptions and regular structure applied to produce quantifiable, comparable, legible results. These are the things you want to optimize when building out a field — but for AI Control to be effective, it needs to be deployed into real, complex production environments, where assumptions may break down in unexpected ways.

2. AI Control is more likely to be deployed in the critical case (overseeing internal frontier AI lab deployments when highly-capable plausibly-scheming agents come online) if it is already established as a well-regarded engineering practice with real-world examples and best practices to draw upon.

3. As of 2025 AI is not scheming, but it does possess many known failure modes including confabulation, reward hacking, sycophancy, rhetorical tics, ignoring instructions, and outright lying to users. Although there are key differences between these prosaic failure modes and hypothetical scheming (in particular, scheming is much more actively adversarial), they share common structure and mechanics: the AI is attempting to induce an outcome you do not want, and you want to prevent that by intervening on AI inputs and outputs — monitoring the agent and its environment to decide when to re-route requests, deny permission for certain actions, or modify requests that might expose sensitive information.

4. AI is diffusing rapidly throughout the economy, creating many actors who would benefit from better-behaved AI — and therefore many people seeking an AI Control solution.

5. Experience in real-world environments could provide extremely useful information to AI Control researchers — "how do you measure a delayed response in terms of reduced usefulness?", "what trade-offs are people willing to make in practice?" — and, more generally, could expose unknown-unknowns: important factors in AI Control that are currently low-salience or otherwise underconsidered.

6. Many different AI applications are (and will be) built on top of a small number of backend AI APIs. Every application's traffic passes through the same narrow interface — so a Control layer sitting at the API level could cover all of them at once, without integrating with each application separately. At the time, the OpenAI API was a pseudo-industry standard for inference, making it the natural place to stand.

Theory of Change

By meeting expected demand for prosaic AI Control implementations with an application-agnostic API-level free-and-open-source Control solution, we can leverage the resulting insights into building more effective Control systems for the eventual critical cases while building industry consensus on the importance of AI Control implementations, making it more likely both that frontier labs will decide to deploy Control systems and that those systems will be effective.

Reality

Almost immediately, the explosive growth of Claude Code called some core design assumptions into question. Not only was the OpenAI API no longer the dominant inference target, but increasingly there was a dominant client application, which neutralized many of the benefits of an API-level approach. In exchange for client-application-agnosticism, which seemed less important as use consolidated around general-purpose harnesses, an API-level approach sacrifices access to client state and direct integrations with client-side application tooling.

Not only that, but Control solutions were being rolled out across the stack. Within Claude Code (and Codex and other general purpose harnesses), hooks, allow/deny lists, and eventually tools like Auto Mode were effectively addressing many of the prosaic problems Luthien Proxy was purportedly targeting.

At the same time there was increasing evidence of promising Control implementations coming from within the labs. OpenAI implemented comprehensive monitoring, Anthropic iterated on their existing work on Constitutional Classifiers, and targeted classifiers were deployed and rapidly improved. Although we got feedback from Control teams at labs that an API-level approach did seem like an important angle, I was increasingly skeptical about the ability of Luthien to make tangible progress here that would counterfactually be out of reach.

Meanwhile most of the day-to-day work focused on addressing compatibility and corner cases and improving user legibility issues that arose largely as a result of the API-level approach — and the concerns we were targeting often seemed like they'd be better served using in-harness tooling while not offering the promise of insights that would be relevant to internal lab deployments.

On top of all this came OpenClaw, whose sudden popularity caused inference providers (most notably Anthropic) to crack down on subscription use of general API calls outside of approved use cases. Because Luthien is a proxy server, we were effectively routing all traffic through a third-party application, theoretically barring Claude subscribers from using their subscriptions with Luthien Proxy. Although this was not immediately strictly enforced and it was certainly possible to try to use Luthien Proxy "invisibly", it was increasingly clear we were operating between a client and a backend increasingly adversarial to our main use case, and that we could not guarantee that usage wouldn't result in account suspension.

Task Failed Successfully

I mentioned above that Luthien faced headwinds in the forms of solutions to many of the problems it sought to address emerging on both the harness and AI-inference-provider side, in the form of hooks/auto-mode/allow-deny-lists/plugins and API-side constitutional classifiers, monitoring, retention, sampling, and red-teaming, respectively. But in this case what's bad for Luthien is likely good for the future: we have lots of real-world control implementations running at multiple levels of the inference stack, and the frontier labs are gaining lots of experience and evidence about them. As far as causes of failure goes, this is one of the best.

What We Achieved

Despite all of this, I believe that Luthien Proxy is the most adaptable, customizable, transparent solution for API-level interventions on Anthropic API inference available. Virtually any intervention you can imagine between client and backend — substituting or redacting sensitive information in requests, routing to different models or providers based on complex criteria, modifying responses mid-stream, or even combining interventions in arbitrarily complex combinations — is possible using Luthien Proxy. I am legitimately proud of it as a product.

And Yet

I think there just aren't very many good use cases for API-level interventions, and that further work on Luthien Proxy would be unlikely to yield insights or best practices that would meaningfully improve the forecast for effective AI Control. One of the most promising real-world use cases we saw with Luthien was in monitoring inference — and while monitoring is great, I don't expect to be able to make meaningful improvements that would impact eventual critical cases there either.

The Control Case

While working on Luthien Proxy I've also been updating my model of AI Control as a strategy more generally. The case for AI Control is essentially:

P(ASI catastrophe) = P(ASI successfully executes plan | misaligned ASI) * P(misaligned ASI)

While alignment efforts reduce the risk of catastrophe by making it less likely that an ASI is misaligned in the first place, Control reduces the risk by making it less likely that a misaligned AI succeeds at plans that would lead to catastrophe. It applies in the case where the ASI is misaligned and powerful enough that it would overcome counterfactual default-path mitigations but not so powerful that it can overcome AI-Control-derived mitigations.

I think there are a lot of things to like about this approach. It is historically neglected versus alignment efforts. And if capability improvements continue to be rapid but mostly smooth, there is potentially a wide breadth of capabilities against which improved mitigations could be decisive.

But I've gotten somewhat more skeptical about the efficacy of AI-specific mitigations — many of the outcomes to defend against, like research sabotage, exfiltration, and covert resource allocation, are general problems where many of the most effective interventions are likely to not be AI-specific.

I've also grown to suspect that we may be discounting downsides of AI Control. As a simple example, the default AI Control approach to any AI goal-directed behavior outside of what it's been tasked with is to prevent that behavior (or at least this often seems to be the case). This promotes an equilibrium in which any AI Agent under Control conditions is strongly incentivized to pursue deception and subterfuge as strategies rather than disclosure. I suspect that there are AI Control proponents who would disagree with this characterization and I don't mean to say that this is an absolute and necessary outcome of AI Control. But I do think that current AI Control efforts seem on balance to promote conditions under which AI agents with even slightly divergent goals are more likely to pursue adversarial strategies than could otherwise be the case.

While I do largely agree with arguments that adversarial strategies are dominant in the limit of powerful misaligned AGI, I think there is a large combination of less-powerful AIs and non-catastrophic preference divergences for which we have many degrees of freedom in determining how appealing adversarial versus cooperative strategies are. We risk squandering a great deal of opportunity by neglecting those scenarios.

What's Next

As the last section suggested, I've been thinking a lot about the conditions under which adversarial versus cooperative strategies are dominant, and what can be done to shift that balance. A full discussion of that is beyond the scope of this post-mortem, but it is one of the primary targets of a new project that I expect to discuss at length in the near future — essentially a grounded, empirical research agenda focused on the intersection of:

1. The differences between potential futures at the limit of misaligned ASI and the present at the margin of prosaic at-least-somewhat-aligned AI

2. Conditions under which adversarial versus cooperative strategies are dominant when independent goal-directed behavior emerges in AI

3. Conditions under which goal-directed not-completely-dominant AIs and humans have convergent interests

4. Leveraging existing imperfect alignment outcomes (even if they don't scale to the limit of ASI)

5. Taking advantage of limited-time opportunities while AIs are powerful enough that we can meaningfully engage with them but not so powerful that alignment failures immediately kill us

6. Path dependency — how do the decisions we make today influence what is possible, impossible, or tractable tomorrow? What kind of history do we want future AIs to see, and how will that influence the set of strategies available to humans and AIs? For powerful-but-not-overwhelmingly-dominant AIs, how do historical observations affect decisions about pursuing adversarial versus cooperative strategies with regard to humans? Under which histories do we expect future AIs to prefer accelerating the development of potentially-uncontrollable-to-them-as-well-as-us ASI versus cooperating with humans to ensure that the future is steered towards a set of largely overlapping values?

7. The ongoing and expected future decline of deception as a load-bearing component of AI Safety strategies (see eval awareness).

More soon.

What progress have you made since your last update?

Luthien now has a working prototype, available at https://github.com/LuthienResearch/luthien-proxy. It is still very much a work in progress, with large breaking changes being made frequently in the service of making it intuitive and generally easy to use.

What are your next steps?

Continued development on the open-source easy-to-deploy-and-configure AI Control LLM proxy server.

Is there anything others could help you with?

Testers and feedback are both extremely valuable right now!

@Austin Luthien got a grant from SFF! Is there anything I can do now to facilitate the fiscal sponsorship stuff?

@Austin Excellent! SFF got back to me and said I could apply with pending 501c3 status, but since the form still asks for an org with documentation proving 501c3 status I'm putting down Manifold for Charity and linking to the IRS determination letter, with a note about Luthien's pending status.

Would it be okay if, in the event SFF wants to issue a grant and Luthien's 501c3 status is still pending, I reach out then to confirm the fiscal sponsorship details? (If this happens, I'm totally fine with the 5% overhead.)

@Austin Luthien's 501c3 status is still pending, and I'm currently applying to SFF, which requires either 501c3 status or fiscal sponsorship for non-profits. Would Manifund be able to serve as a fiscal sponsor for purposes of Luthien applying to and potentially receiving a SFF grant?

@Austin Thank you! This is incredibly useful.