I'm concerned about conflicts of interests Epoch AI might have, by receiving funding from OpenAI. Please tell me if this is unjustified.
Epoch AI is a leading research institute that develops public knowledge on AI's trajectory through rigorous research, novel benchmarking, and comprehensive data tracking. We seek general support to expand our work informing policymakers, industry leaders, and society on AI development and its implications. Our unique position combining technical depth with accessible communication has made us a trusted source for stakeholders ranging from government officials to tech leaders.
Broadly, our mission is to conduct and disseminate rigorous research on the trajectory and impact of artificial intelligence, to enable better policy and decision-making. With additional funding, we aim to expand three key initiatives:
Data Curation: Building on our successful AI model and hardware tracking to cover AI clusters and companies, launching weekly data insights for timely analysis.
Capability Measurement: Growing our AI Benchmarking Hub through independent model evaluations and novel benchmarks like FrontierMath.
Economic Modeling: Developing a framework combining AI scaling laws with growth theory to analyze automation impacts and compute scaling.
We'll achieve these through strategic hiring of technical talent, maintaining our commitment to rigorous analysis, and expanding our successful track record of creating accessible, high-impact content.
We're seeking $10M over two years to scale our impact:
$2M/year for the data program (tracking AI systems, hardware, clusters)
$2M/year for AI benchmarking (capability measurement, new benchmark development)
$6M for team expansion including an AI Data Lead, research engineers, and operations staff
Reaching 30 full-time employees while maintaining project-specific contractor relationships
Our 23-person team (https://epoch.ai/about) has demonstrated strong execution in 2024:
Published groundbreaking benchmarks (FrontierMath) with 70+ mathematicians
Released major reports cited by tech leaders and government officials
Published in top venues (ICLM, NeurIPS 2024)
Grew audience from 2K to 15K Twitter followers
Attracted 360K website users
Successfully scaled from 10 to 23 full-time employees this year
Talent Competition: The competitive market for AI researchers could dampen our capacity to attract and retain talent
Data Access: Limited visibility into private AI development could affect our tracking capabilities
Roughly 75% of our funding comes from Open Philanthropy. The rest comes from individual donors, organizations such as Survival and Flourishing Fund, or from service agreements with collaborators on projects aligned with our mission.
Kabir Kumar
20 days ago
I'm concerned about conflicts of interests Epoch AI might have, by receiving funding from OpenAI. Please tell me if this is unjustified.
Kabir Kumar
20 days ago
Additionally, I'm confused as to why this funding wasn't disclosed clearly and openly from the start.
Jaime Sevilla
16 days ago
@KabirKumar Hey! Here Jaime, director of Epoch. For context on the situation with OpenAI, please read this blogpost which hopefully will clarify what happened and our current relation with OpenAI.
https://epoch.ai/blog/openai-and-frontiermath
The TL;DR is that they funded and have exclusive access to FrontierMath, a math benchmark.
Going forward, we plan to continue collaborating with OpenAI and other labs on creating other benchmarks, and possibly other projects that advance our mission of informing the public about the trajectory of AI.
Ryan Kidd
16 days ago
@KabirKumar If concerned, surely it's good to fund Epoch via Manifund to reduce their dependence on OpenAI?
Lun
about 1 month ago
Token donation to signal support. (Somewhat amusing juxtaposing with Neel's much larger token regrant.)
Neel Nanda
about 1 month ago
I'm re-granting because I've been very impressed with the quality of Epoch's work. It's clearly far and away better than any competition when it comes to actually understanding what's going on with AI (and being publicly communicated), and several people I know who are part of Epoch strike me as highly competent. I'm also a fan of their new weekly newsletter. I haven't carefully checked the details of much of my work myself, but the one example where I thought I found an error (using total rather than active parameters of an MoE model), they then produced a newsletter with a compelling argument for why their way made more sense, which I was impressed by.
I see the main theory of impact of Epoch as broadly helping key decisionmakers in society (policy-makers, businesses, the AI safety community, etc) be well informed about AI: what has already happened, what is happening, what will happen. I'm a bit less confident in how well their projects do on the dissemination front. I see them go viral fairly often, though I am totally in a massive bubble. I've seen some scattered examples like Satya Nadella having Epoch graphs in a slide deck, which seem pretty promising!
Is this theory of impact important?
I think that AI is clearly a big deal, at this point, having an important impact on society, and expect this to accelerate. In addition to the big issues of misuse and misalignment, there's also just a lot of social and policy change that needs to happen to handle this well. By default, society changes slowly, and tech policy often makes little sense. High quality information is far from sufficient to fix this, but it helps!
I have some concern that this also has negative externalities by increasing the number of people who realise AI is a big deal and decide to race (in a similar sense to how, eg, scaling laws got more people realising the potential of AI). But my guess is that this isn't that big a deal. I think that "AI is a big deal" is pretty widely believed at this point, so that ship has largely sailed. Another concern is that this increases competitive pressures by making it clearer to people when they're losing the race. But on the other hand, this helps the person in first place realise it, which is high value (see eg the Cold War "missile gap"). And I expect key actors like AGI labs already have much better awareness of what's going on than most, so Epoch adds less value to them.
Overall, this isn't my area of expertise so I'm not confident, but my best guess is that Epoch is doing great work.
I'm only donating a token amount as I don't think this is my comparative advantage as a funder, but I would encourage others to donate, and want Epoch to have as much funding as they can productively use.
Oliver Habryka
about 1 month ago
Agree on this. I do think the strongest argument against funding Epoch for most people on Manifund is that it can and probably will be supported aggressively by existing large funders. Evidence that this isn't the case would update me a lot on more funding to Epoch being valuable.
Meeri
10 days ago
@RyanKidd They clearly state that FrontierMath was a project commissioned by OpenAI. https://epoch.ai/blog/openai-and-frontiermath (which makes me think that your story isn't very accurate, let me know if I'm confused)
More to the point, it seems very backwards to me to take Epoch AI secretly doing commissioned work for OpenAI without letting the contributors know as a reason to give more funding for Epoch AI. Some thoughts
"Roughly 75% of our funding comes from Open Philanthropy. The rest comes from individual donors, organizations such as Survival and Flourishing Fund, or from service agreements with collaborators on projects aligned with our mission." They're choosing not to mention industry funding on their Manifund. I feel like they're not very trustworthy in being transparent in their communication. Are they 75% OpenPhil funded?
Tamay has tweeted that the holdout set was independently funded (https://x.com/tamaybes/status/1870047603901247876) but Epoch AI says that OpenAI owns the holdout set (https://epoch.ai/blog/openai-and-frontiermath). This is all very confusing (and again, we don't fully know about Epoch AI funding) but I feel like if your story of more Epoch AI funding reducing reliance on commercial partners worked well, then nothing that was publicly communicated to be independently funded would ever be later revealed as owned by OpenAI.
They were happy to do a big project that was secretly commissioned by OpenAI, without communicating this to the 70+ mathematicians. In this comment section Jaime Sevilla says that going forward, they plan to continue collaborating with OpenAI. What in this situation makes you think they have intention to reduce industry collaboration if they get bigger?
If a project would otherwise be funded by OpenAI, then you're probably not having great counterfactual impact by donating there.
Ryan Kidd
10 days ago
@Meeri At this point I don't feel qualified to comment and I defer to @Jsevillamol.