I decided not to fund this, but accepted Matthew into MATS instead.
@RyanKidd
Co-Director at MATS, LISA
https://www.linkedin.com/in/ryan-kidd-1b0574a3/This is a donation to this user's regranting budget, which is not withdrawable.
$4,000 in pending offers
Currently growing the AI alignment research field at ML Alignment & Theory Scholars Program (MATS) and the London Initiative for Safe AI (LISA). Previously, I completed a PhD in Physics at the University of Queensland and ran an Effective Altruism student group for ~3 years.
My ethics are largely preference utilitarian and cosmopolitan. I'm deeply concerned about near-term x-risk and safeguarding the long-term future. I see Manifund as an opportunity to fund public benefit research into high-impact cause areas that lack adequate incentive mechanisms.
My grantmaking prioirites include:
Neglected AI safety interventions with a good safety/capabilities trade-off (i.e., R&D that might not be near-term commerically beneficial, but "raises the AI safety waterline");
Democratic governance mechanisms for monitoring and regulating dangerous "black-ball" technology, especially in AI and synthetic biology;
Building "defence in depth" and "civilization resilience" to systemic risk factors, especially in regards to AI accident/misuse, pandemics, great power war, and totalitarianism.
In addition to general grantmaking, I have requests for proposals in the following areas:
Funding for AI safety PhDs (e.g., with these supervisors), particularly in exploratory research connecting AI theory with empirical ML research.
An AI safety PhD advisory service that helps prospective PhD students choose a supervisor and topic (similar to Effective Thesis, but specialized for AI safety).
Initiatives to critically examine current AI safety macrostrategy (e.g., as articulated by Holden Karnofsky) like the Open Philanthropy AI Worldviews Contest and Future Fund Worldview Prize.
Initiatives to identify and develop "Connectors" outside of academia (e.g., a reboot of the Refine program, well-scoped contests, long-term mentoring and peer-support programs).
Physical community spaces for AI safety in AI hubs outside of the SF Bay Area or London (e.g., Japan, France, Bangalore).
Start-up incubators for projects, including for-profit evals/red-teaming/interp companies, that aim to benefit AI safety, like Catalyze Impact, Future of Life Foundation, and YCombinator's request for Explainable AI start-ups.
Initiatives to develop and publish expert consensus on AI safety macrostrategy cruxes, such as the Existential Persuasion Tournament and 2023 Expert Survey on Progress in AI (e.g., via the Delphi method, interviews, surveys, etc.).
New nonprofit startups that aim to benefit AI safety.
Ryan Kidd
27 days ago
I regranted an additional $2k to let the organizers launch the basic event, as per Grace's comment.
Ryan Kidd
about 1 month ago
Update: we recently published a blog post summarizing our takes on talent needs of technical AI safety teams based on 31 interviews with key figures in AI safety, including senior researchers, organization leaders, social scientists, strategists, funders, and policy experts.
Ryan Kidd
about 1 month ago
Update: we recently published our Winter 2024-25 Retrospective: https://www.lesswrong.com/posts/Z87fSrxQb4yLXKcTk/mats-winter-2023-24-retrospective
Ryan Kidd
about 2 months ago
Update: MATS is no longer in need of additional funding for our Summer 2024 Program. We are still accepting donations towards our Winter 2024-25 Program, however!
Ryan Kidd
about 2 months ago
I'll likely regrant to this project because I think CAIS is great, but I'll first look for projects where my grants funge less with Open Phil, SFF, Longview, LTFF, etc.
Ryan Kidd
about 2 months ago
Dan Hendrycks' AI safety textbook seems great, but principally serves as an introduction to the field, rather than an in-depth overiew of current technical AI safety research directions, which is the intent of this project. Periodically updated "topics courses" could serve as an equivalent source of value, but these might be bound to particularly universities and updatable on a slower timescale than an online textbook. I'm also enthused by Markov's plans to eventually integrate interactive content and live content from sources like Metaculus, Our World in Data, Stampy, and more.
I believe that the AI safety research field should grow 10-100x over the next 10-20 years and AI safety student groups should be a strong driver of this growth. Currently, I think AI safety student groups need more "plug-and-play" curricula to best prepare members for progression into research, engineering, and policy roles, especially at universities without dedicated AI safety courses like that based on Hendrycks' textbook. I think BlueDot Impact's AI Safety Fundamentals courses are great, but I don't see why BlueDot and CAIS should be the only players in this space and think there is some benefit from healthy competition/collaboration.
Charbel has road-tested content from the early stages of this textbook project with several AI safety university groups and courses with apparently good feedback.
I've been impressed with Charbel's LessWrong posts and nuanced takes on AI safety research agendas.
The online version of the textbook will be free and open-source (MIT License), which I think is important for introductory AI safety fieldbuilding materials to be maximally impactful.
I think that the optimal form of this project is a continually updated online resource that periodically integrates new papers and research paradigms and therefore this project will eventually need long-term funding and permanent home. However, I believe that my grant will greatly assist Charbel and Markov in producing a proof-of-concept sufficient to secure long-term funding or institutional support. Additionally, the textbook MVP seems likely to be high-value for the near-term regardless of whether the project continues. Lastly, if the textbook is high-value and Charbel and Markov are unable to secure long-term funding, I'm sure it will be useful for established curriculum developers like BlueDot Impact.
I wonder if this project should actually be converted into a wiki once the MVP is developed. Markov has previously worked with Stampy and has mentioned that they might want to integrate some Stampy articles into the online textbook. However, even if this project is ideally a wiki, building a viable MVP seems crucial to securing long-term funding and core content for iterating upon.
I don't know Markov or the proposed editor, Professor Vincent Corruble, very well, which slightly decreases my confidence in the textbook quality. However, Markov comes highly recommended by Charbel, has previously worked as at Rational Animations in charge of AI safety, and has produced good-according-to-me content for the textbook so far. Professor Corruble is an Associate Professor Sorbonne Université and a UC Berkeley CHAI affiliate, which indicate he has the technical expertise to oversee the computer science aspects of the the textbook. I additionally recommend that Charbel and Markov enlist the support of further editors with experience in AI safety strategy and AI governance, as I believe these are critical aspects of the textbook.
I chose to donate enough to fund the minimum amount for this project to proceed because:
I want full-time work on this textbook to commence immediately to minimize its time-to-impact and Markov is unable to do this until he receives confirmed funding for 6 months;
I think it is relatively unlikely that this project will be funded by Open Philanthropy or the LTFF and I have greater risk tolerance for projects like this;
I have 5x the grant budget I had last year and I think this project is probably more impactful than I would have considered necessary for a counterfactual $7.8k regrant made last year based on the projects I funded;
I didn't give more than the minimum amount as I feel my marginal funding is high-value for other projects and I think Charbel and Markov can likely secure additional funding from other sources (including other Manifund regrantors) if necessary.
I don't believe there are any conflicts of interest to declare.
Ryan Kidd
2 months ago
Update: thanks to your donations, we were able to support an additional 8.5 scholars in the Winter 2023-24 Program, at an ex post cost of $22.4k/scholar! Thank you so much for your contributions to the field of AI safety :)
We are currently fundraising for our Summer 2024 Program and again expect to receive less funding than our ideal program. We can support marginal scholars at a cost of $24.4k/scholar. We currently have 1220 applicants for Summer 2024 and expect to accept ~3-5% (i.e., MIT's admissions rate). Given the high calibre of applicants and mentors, we would love further funding to support additional scholars!
We have announced the following mentors and hope to announce more as we confirm additional funding: https://docs.google.com/document/d/1sDnD9Igr3gkWX-N_l9W8itVBpqx-pChh-61atxGYkPc/edit
Ryan Kidd
6 months ago
I think that there should be more AI safety organizations to: harness the talent produced by AI safety field-building programs (MATS, ARENA, etc.); build an ecosystem of evals and auditing orgs; capture free energy for gov-funded and for-profit AI safety organizations with competent, aligned talent; and support a multitude of neglected research bets to aid potential paradigm shifts for AI safety. As an AI safety organization incubator, Catalyze seems like the most obvious solution.
As Co-Director at MATS, I have seen a lot of interest from scholars and alumni in founding AI safety organizations. However, most scholars do not have any entrepeneurial experience and little access to suitable co-founders in their networks. I am excited about Catalyze's proposed co-founder pairing program and start-up founder curriculum.
I know Kay Kozaronek fairly well from his time in the MATS Program. I think that he has a good mix of engagement with AI safety technical research priorities, entrepeneurial personality, and some experience in co-founding an AI safety startup (Cadenza Labs). I do not know Alexandra or Gábor quite as well, but they seem driven and bring diverse experience.
I think that the marginal value of my grant to Catalyze is very high at the moment. Catalyze are currently putting together funding proposals for their first incubator program and I suspect that their previous Lightspeed funding might run low before they receive confirmation from other funders.
Alexandra and Kay do not have significant experience in founding/growing organizations and none of the core team seem to have significant experience with AI safety grantmaking or cause prioritization. However, I believe that Gábor brings significant entrepeneurial experience, and Jan-Willem and I, as advisory board members, bring significant additional experience in applicant selection. I don't see anyone else lining up to produce an AI safety org incubator and I think Alexandra, Kay, and Gábor have a decent chance at succeeding. Regardless, I recommend that Catalyze recruit another advisory board member with significant AI safety grantmaking experience to aid in applicant/project selection.
It's possible that Catalyze's incubator program helps further projects that contribute disproportionally to AI capabilities advances. I recommend that Catalyze consider the value alignment of participants and the capabilities-alignment tradeoff of projects during selection and incubation. Additionally, it would be ideal if Catalyze sought an additional advisory board member with significant experience in evaluating dual-use AI safety research.
There might not be enough high-level AI safety research talent available to produce many viable AI safety research organizations right away. I recommend that Catalyze run a MVP incubator program to assess the quality of founders/projects, including funder and VC interest, before investing in a large program.
Alexandra said that $5k gives Catalyze one month of runway, so $15k gives them three months runway. I think that three months is more than sufficient time for Catalyze to receive funding from a larger donor and plan an MVP incubator program. I don't want Catalyze to fail because of short-term financial instability.
I am an unpaid advisor to Catalyze. I will not accept any money for this role.
Kay was a scholar in MATS, the program I co-lead. Additionally, I expect that many potential participants in Catalyze's incubator programs will be MATS alumni. Part of MATS' theory of change is to aid the creation of further AI safety organizations and funders may assess MATS' impact on the basis of alumni achievements.
Catalyze wants to hold their incubator program at LISA, an office that I co-founded and at which remain a Board Member. However, I currently receive no income from LISA and, as a not-for-profit entity, I have no direct financial stake in LISA's success. However, I obviously want LISA to succeed and believe that a potential collaboration with Catalyze might be beneficial.
My donation represents my personal views and in no way constitutes an endorsement by MATS or LISA.
Ryan Kidd
8 months ago
Update update: Several more awesome mentors have come forward and we now are funding constrained again for Winter!
Ryan Kidd
8 months ago
Update: we don't appear to be funding constrained for Winter, but will continue accepting donations for our Summer 2024 Program!
Ryan Kidd
9 months ago
Developmental interpretability seems like a potentially promising and relatively underexplored research direction for exploring neural network generalization and inductive biases. Hopefully, this research can complement low-level or probe-based approaches for neural network interpretability and eventually help predict, explain, and steer dangerous AI capabilities such as learned optimization and deceptive alignment.
Jesse made a strong, positive impression on me as a scholar in the SERI MATS Winter 2022-23 Cohort; his research was impressive and he engaged well with criticism and others scholars' diverse research projects. His mentor, Evan Hubinger, endorsed his research at the time and obviously continues to do, as indicated by his recent regrant. While Jesse is relatively young to steer a research team, he has strong endorsements and support from Dan Murfet, David Krueger, Evan Hubinger, and other researchers, and has displayed impressive enterpeneurship in launching Timaeus and organizing the SLT summits.
I recently met Dan Murfet at EAGxAustralia 2023 and was impressed by his research presentation skills, engagement with AI safety, and determination to build the first dedicated academic AI safety lab in Australia. Dan seems like a great research lead for the University of Melbourne lab, where much of this research will be based.
Australia has produced many top ML and AI safety researchers, but has so far lacked a dedicated AI safety organization to leverage local talent. I believe that we need more AI safety hubs, especially in academic institutions, and I see Timaeus (although remote) and the University of Melbourne as strong contenders.
Developmental interpretability seems like an ideal research vehicle to leverage underutilized physics and mathematics talent for AI safety. Jesse is a former physicist and Dan is a mathematician who previously specialized in algebraic geometry. In my experience as Co-Director of MATS, I have realized that many former physicists and mathematicians are deeply interested in AI safety, but lack a transitionary route to adapt their skills to the challenge.
Other funders (e.g., Open Phil, SFF) seem more reluctant (or at least slower) to fund this project than Manifund or Lightspeed and Jesse/Dan told me that they would need more funds within a week if they were going to hire another RA. I believe that this $20k is a high-expected value investment in reducing the stress associated with founding a potentially promising new AI safety organization and will allow Jesse/Dan to produce more exploratory research early to ascertain the value of SLT for AI safety.
I have read several of Jesse's and Dan's posts about SLT and Dev Interp and watched several of their talks, but still feel that I don't entirely grasp the research direction. I could spend further time on this, but I feel more than confident enough to recommend $20k.
Jesse is relatively young to run a research organization and Dan is relatively new to AI safety research; however, they seem more than capable for my level of risk tolerance with $20k, even with my current $50k pot.
The University of Melbourne may not be an ideal (or supportive) home for this research team; however, Timaeus already plans to be somewhat remote and several fiscal sponsors (e.g., Rethink Priorities Special Projects, BERI, Ashgro) would likely be willing to support their researchers.
I chose to donate $20k because Jesse said that a single paper would cost $40k (roughly 1 RA-year) and my budget is limited. I encourage further regrantors to join me and fund another half-paper!
Jesse was a scholar in the program I co-lead, but I do not believe that this constitutes a conflict of interest.
Ryan Kidd
11 months ago
@alenglander, when do you expect to hear back from the LTFF? Was the Nonlinear Network funding successful?
For | Date | Type | Amount |
---|---|---|---|
Evaluating the Effectiveness of Unlearning Techniques | 4 days ago | project donation | 10000 |
MATS Funding | 5 days ago | project donation | +2000 |
MATS Funding | 28 days ago | project donation | +1000 |
AI Safety Textbook | about 2 months ago | project donation | 39000 |
MATS Funding | about 2 months ago | project donation | +400 |
Manifund Bank | about 2 months ago | withdraw | 81040 |
MATS Funding | 2 months ago | project donation | +1040 |
MATS Funding | 2 months ago | project donation | +80000 |
Manifund Bank | 3 months ago | deposit | +250000 |
AI Safety Research Organization Incubator - Pilot Program | 6 months ago | project donation | 15000 |
Help Apart Expand Global AI Safety Research | 6 months ago | project donation | 5000 |
Manifund Bank | 6 months ago | withdraw | 190178 |
AI Policy work @ IAPS | 6 months ago | project donation | 5000 |
Cadenza Labs: AI Safety research group working on own interpretability agenda | 6 months ago | project donation | 5000 |
MATS Funding | 6 months ago | project donation | +14000 |
MATS Funding | 6 months ago | project donation | +134 |
MATS Funding | 6 months ago | project donation | +1211 |
MATS Funding | 6 months ago | project donation | +17533 |
MATS Funding | 7 months ago | project donation | +6000 |
MATS Funding | 7 months ago | project donation | +500 |
MATS Funding | 7 months ago | project donation | +150000 |
MATS Funding | 7 months ago | project donation | +300 |
MATS Funding | 7 months ago | project donation | +500 |
Scoping Developmental Interpretability | 9 months ago | project donation | 20000 |
Manifund Bank | 11 months ago | deposit | +50000 |