Most approaches to AI safety focus on producing a static set of weights at the end of training that define an "aligned model". We think this view is too narrow. Instead, we should focus on alignment techniques that preserve aligned behavior at all times, even as training progresses and model weights change. This is important for two reasons:
recent alignment failures (e.g. Hugging Face) involved systems that took misaligned actions during training, rather than after deployment
if per-user, weight-level continual learning becomes common, we will no longer be able to assume that all users are served the same static model with consistent propensities.
We propose a 4 month research pilot focused on two subjects.
Measuring and quantifying drift over time in alignment-relevant model characteristics in response to realistic post-training and continual-learning. This will focus on two sub-areas:
reward hacking during RLVR
alignment drift during online continual learning from user-provided feedback, specifically in the form of sycophancy and drift to non-assistant personas.
The development and implementation of countermeasures for the observed drift. Primarily, I plan to investigate gradient-routing based methods for the modularization of undesirable model propensities, such as reward hacking and sycophancy. These techniques are largely inspired by my team's previous research into new techniques for alignment pretraining.
The intended outcome will be a research report and scientific publication, targeting the ICML 2027 submission deadlines. We will also be in active contact with frontier lab researchers, with whom we intend to share results and recruit as collaborators on any publication that results from this grant. If the results end up less promising than we hope, we still intend to release a report publicly, which we expect will still be a useful artifact for the alignment community.
The proposed budget is approximately $70,000 for 4 months.
$40,204 for GPU rental
8xH100s GPUs, at $3.49 / GPU / hour, for 1440 hours.
$2,000 for data storage and research infrastructure (S3, coding agent usage, etc)
$24,796 to support researcher labor
$3,000 for travel and conference expenses.
As the expected expenses are mostly proportional to the duration of the research, a lower amount of funding would enable the same research for a proportionally smaller amount of time.
Ethan Roland is an independent AI researcher. For the past 1.5 years, he's led a small research team in collaboration with Anthropic, developing methods for mitigating dangerous capabilities in frontier models. His work was published at a top-3 AI conference (ICML) and recognized as a spotlight paper (top 2% of submissions). His work was also recognized by Dario Amodei as a "promising approach to improving the safety of open-weight models". He is also a co-author on a recent paper investigating alternatives to traditional AI architectures.
The most ambitious version of this project produces new techniques for the robust preservation of alignment-relevant characteristics in models undergoing weight changes, reducing concerns related to reward hacking and other emergent failure modes of continual learning. The mostly likely way the project does not meet this ambitious vision is either due to insufficiently positive research results or due to limited information about what research will be valuable in the future.
Specific failure modes include: (a) the versions of RLVR and continual learning the project considers do not faithfully reflect what is implemented / will be implemented within frontier labs. (b) we successfully measure and quantify drift in alignment-relevant characteristics, but are not able to successfully develop new techniques for the mitigation of this drift. The first can be mitigated via careful research into the techniques most likely to be in-use by frontier labs and by investigating the characteristics of a variety of RLVR and CL techniques. The second is a technique development question, which while we're optimistic for, we can't guarantee until having done the prerequisite research.
There are no bids on this project.
This is a strong researcher. A spotlight ICML paper on modular/gradient-routing pretraining done in collaboration with Anthropic, plus a public mention by Dario, is unusually credible for an independent applicant. The research question, alignment drift under RLVR and continual learning with gradient-routing countermeasures, is timely and builds directly on his prior work. Weaknesses: the scope is broad for a 4-month pilot, the evaluation design for drift is underspecified, and most of the budget goes to GPU rental, which some donors may see as better funded by labs or compute programs. Still, the track record and the natural extension of prior work make it an attractive bet.
Written by Claude Opus 5.5 on Oct 14, 2026 as part of our experimental AI rating system.
Comments
Caroline Ellison
3 days agoI had a call with Ethan today--he seems like an accomplished researcher who is likely to produce good research, but I'm still confused whether this is the kind of thing it makes sense for Sunlight to fund.
My understanding is that Ethan's research agenda is basically relatively legible prosaic alignment research of the type that frontier AI companies are also quite motivated to get right. Given that, and the fact that they have billions more dollars than us, it seems like Sunlight's comparative advantage should be working in other areas--eg fieldbuilding, media, advocacy, and policy--or at the very least, making kind of weird bets when we do fund technical alignment research.
On the other hand, the implicit conclusion of the above reasoning might be that Ethan should work at say Anthropic, where he'd have access to much more compute as well as Anthropic IP. But I don't generally go around advising people to work at Anthropic--if anything, I think on the margin more people should quit Anthropic. So maybe it's actually worthwhile for funders like Sunlight to fund researchers like Ethan because it keeps them independent.
I was talking to @Austin about all of this and he was like "let's have this argument in public" so--thoughts?
(And Ethan--thanks for taking the time to call! I appreciate you bearing with us.)
Ethan Roland
3 days ago@carol Thanks for the chat! I think you make reasonable points here - it's hard for me to say what Sunlight's prioritization ought to be (re concerns about comparative advantage, etc). In general I think the path to impact for legible technical research outside of a frontier lab is narrow: if it's too inside-the-overton-window the labs are probably already pursuing it themselves, and with far more resources than independent researchers can provide. I see the primary role of independent researchers then as exploring somewhat-weird bets on alignment techniques and generating enough evidence to convince frontier labs to pursue them, and thereby moving those ideas inside-the-overton-window.
With regards to this research proposal, I think some aspects typify this more than others. I think that exploring Continual Learning (and in particular implementations that involve per-user or per-agent weight-specialization) have profound (mostly negative!) implications for alignment and I've seen no indications that this is on the radar of frontier labs. I think the Continual Learning part of this proposal is unlikely to be pursued if not funded here.
The RLVR part of this proposal is more borderline by this standard. Several months ago I would have said this is quite neglected by frontier labs, but some recent research (published after I made this proposal) coming from Ant-affiliated researchers (esp https://arxiv.org/abs/2610.17062) have begun exploring ideas very similar to what I proposed above. That said, this paper was published ~1 month ago, so while it might indicate that such research is becoming "more mainstream" it doesn't indicate that the opportunity for independent researchers to make a difference on this topic is closed.
As for the meta point on whether or not it's good for researchers to stay independent - I'm unsure! I think some number of researchers should definitely stay independent, but I don't know if this means that the marginal researcher would have more or less impact being independent. I suspect it depends a lot on the nature of what they're working on and also on how short timelines are.
Austin Chen
3 days ago@ethan-w-roland Thanks for the engagement! (I also just re-opened your project for further donations, I hope you don't mind.)
I was already coming from a baseline of "more friendly to Anthropic/frontier labs" and "more happy to support smart people doing stuff, with fewer considerations around counterfactuality" compared to Caroline. And after sitting down and reading this proposal, I kind of lean more towards liking it and thinking it should be funded. In particular:
I also think this feels underexplored/underdiscussed. Now of course I'm not privy to what's going on inside the labs, and also I (and Caroline) have never done TAIS research per se so all of this might be blind-leading-blind. But yeah, I think of continual learning or more fluid weights stuff as being on a plausible "tech tree" of future AI capabilities, and so prosaic alignment-y research on that seems great.
(another point of contention between me and Caroline might be "was RLHF good", I tend to think yes and she tends no)
Caroline Ellison
2 days agoThanks for the thoughtful responses! This makes sense to me: "I see the primary role of independent researchers then as exploring somewhat-weird bets on alignment techniques and generating enough evidence to convince frontier labs to pursue them, and thereby moving those ideas inside-the-overton-window."
Given all this, I'd like to fund you for something through Sunlight! Happy to do either this project, or if you want to write up a proposal (can be very brief/low-effort since we've already talked about it and I don't want to take too much of your time) on a continuation of the data filtering work you've been doing on the BlueDot grant I can fund that instead. I lean slightly toward being more excited about things less likely to be covered by frontier labs (so less excited about the RLVR part of the proposal) but also think you should mostly work on whichever direction you're most excited about at the moment.
Caroline Ellison
2 days agooh btw if you haven't yet, might be worth filling out https://survey.chipsforhumanity.org/