I know orpheus and am excited for these projects, since he has a proven track record already. I also have had personal use with his projects like AI Safety Events - https://www.aisafety.com/events-and-training

I am excited to post an update on the successful conclusion of the workshop -

https://docs.google.com/document/d/104yIBUsUdcdcTWk5rKUfKKyxuS8dx1VKv7IZMolROS0/edit?usp=sharing

I'd love to see this happen! I enjoyed being there for the previous iterations in Bangalore, India

What progress have you made since your last update?

On April 6th I signed the grant agreement.

I filled out the withdrawal form to get the funds on May 24th 2024.

Manifund sends me money on 21st June via Mercury and it reaches my domestic Indian bank account on 26th June.

So 1000 USD would be 81,873 INR, I spent a total of 118,740 INR accounted for in the invoices attached - https://www.notion.so/explict/Manifund-1-HPMOR-339c1429d14c4d1e9799a6e384e7b042?pvs=4

Initially I spent 360 to 425 per book but the quality was subpar. I distributed 15 books at an AI safety meetup organized by Axiom Futures event, where we had alignment researchers meet others (PhD students, entrepreneurs, etc) who are interested in AI Safety in Bangalore.

101 books were distributed in IIIT Delhi and IIT Delhi as part of a talk I gave at both institutes where I talked about risks from increasingly generally intelligent systems, an overview of the current landscape and how there were programs like MATs, ARENA, hackathons organised by APART which these students could participate in

What are your next steps?

Now 95 copies for 799 INR each are coming, delivery should happen in a week. I hope to distribute them at the AI safety hackathon happening in the first two weekends of Oct 2024 and at the EAGx Bangalore happening in the 3rd weekend.

Is there anything others could help you with?

I would like feedback/comments on how best to measure impact, and my progress so far. What questions should the survey have?

https://docs.google.com/forms/d/1M6Yo55xoYUwrPNbpzgDOv-ALrZ0MuMP36MMFfIKo1V4/edit

For now I have gotten almost 60% responses on the QR code but now the plan is to make it mandatory to fill up form on receiving the copy upfront. Then after 2 months I plan to reach out to them to see if they read the book, lent it to anyone else, and or would like the volume 2.

I hope others also see this project and contribute financially or in terms of time and attention to this project.

This is one of the most important projects in Cyborgism alignment agenda, we need to observe multiple models prompting each other to push the limits of what is possible and even likely with these systems, meaning we build better tacit models of them, we are slightly more prepared.

retroactive funding is amazing and needs more love!

I am very excited about Sahil's research agenda - High Actuation Spaces. I think it is worth investing in.

I really enjoy the posts Kaarel writes on lesswrong, I have met him in person and am impressed by his intelligence and work ethic. I believe developmental interpretability is definitely worth more attention and funding. It is exciting to imagine where this will lead.

I really like this idea. I want this to happen in some form, so I hope this moves the bar on this.

@Canadiak I am already giving links out to people I meet and trying to get people to read the soft copy. I believe there is not a lot of margin left there because people who will check it out with low activation energy already have.

The people who prefer hard copies, who are likely to read a book they won in a competition or feel like they earned it, are more likely to complete it and even share it with friends and family.

I think doing a book review competition for the first 10 chapters is still a good idea. I would like to think more about this because I really don't want people to engage with the book like it is a chore they are finishing just to get the prize.

There are examples of my past where I did not really engage with my social science textbook with curiosity because my teacher made it seem like reading it was a task I had to complete to answer a specific set of questions. So I was outcome oriented in how I read it.

It was more than a decade later when I reread those school textbooks with curiosity and openness, I discovered they were actually very interesting and covered history, geography in a fun way.

So I believe the attitude also matters, especially for younger kids where things things are implicitly decided based on how they discover the book and other contextual details.

@ms Thanks for the positive signal Saul.

I also can relate to the positive points you mentioned. I think it is a cost-effective use of resources and a surprising number of people seem to have found the alignment problem through the fanfiction pipeline.

Re: the right hands part,

I believe you mean the right people would be people who are likely to enthusiastically enjoy the book, and have the ideas in it affect them

Even if they do not become alignment researchers, I still think it can help them to discover works like Replacing Guilt, and Sequences and apply more rational thinking in their lives. It also helps to find that this community exists out there, it feels nice to belong.

To increase the odds these books are not gathering dust somewhere and are actually read, it helps to target places where the right people are higher in density. I think places where people engage in intellectually stimulating activities, where abstract meta-thinking skills are useful are good places.

I am currently planning to distribute 70% at events held in top universities in India where the competitive exam vets for problem solving skills and gathers smart kids in one place. Then developer events where agentic, curious, altruistic people come to talk about open source software, AI, etc. There are other places like meetups for quantified self, productivity or note-taking apps, which I think are really great spaces.

The remaining 30%, I am planning to go to schools, target younger people who are unlikely to exhibit clear signs of potential but still I would look for agentic behaviour in them by asking the class who the curious kid is, who reads widely, a generalist, etc.

I am excited to talk to Mikhail, I am quite open to adjusting my plans with new information or ideas.

Re: are these the sorts of people that we would want working on alignment?

Before the overton window shifted so much, it made sense to rely on some gate keeping to ensure high signal to noise ratio in alignment research. But I think now we need more robust mechanisms to decide if someone is contributing to the solution or not. The selection pressures by prematurely deciding what kind of people can work on it is likely to yield unacceptable false negatives.

My model is also that we don't have sufficient confident in existing alignment directions that we can commit resources towards exploit and select for people who are making progress in them. Rather we need to still fund explore type research to hopefully make some tractable progress in time.


> are there other books/series/etc that would get more impactful people engaged?

To get people more quickly engaged I would use superintelligence or Brian's Alignment Problem. If enough funders are excited about that I would be happy to use part of the funds for those books too.

HPMOR is not a quick way to get people into alignment if that is the goal. What HPMOR does is get people interested in the rationality community and practice thinking more carefully about risks from such systems. It helps in a more indirect way, communicating the vibe of thinking a certain way.

It will definitely nerd snipe a certain kind of person and the question of if hpmor is worth funding as opposed to other books would depend on what impact means, are these alignment researchers mentioning hpmor doing impactful work? or are they maybe feeding into some sort of deference cascade?

Personally I believe it is a net positive and so I am happy to work on this project but I don't think the answer is cut and dried. People can have well informed views that think hpmor would not have the right kind of selection pressure.

Re: does HPMOR also unnecessarily alienate people to alignment?

I think this also links to the idea of HPMOR being given as a way to push people into alignment. The downside risk of it alienating people comes from feeling manipulated into a certain frame.

This is why I think it is useful to focus on HPMOR as a tool to raise the sanity waterline or increase awareness about ways of thinking. This might lead to the readers being curious about the generator of the work, reading other works, thinking more clearly about AI safety, and wanting to contribute to the field as a side effect. But that would come from within and I don't think it can or should be forced.

It is also true that this pipeline will lead people to discover alignment through the MIRI worldview first and that might bias them. But there is a lot of dissent and other opinions in this space, so even if they find the problem like this, they can engage with and criticize those framing.

But all that risks of alienation comes only if people read this piece of fan fiction, enjoy it enough to go down the rabbit hole of EY's other work. By itself, I do not think the book can piss off anyone.

Even people who dislike or disagree with EY can still enjoy this book and recommend others read it. I believe art stands separate from the artist at least in this case.

@ms

I had heard that the critical concern JK had was her fans not be exploited using her IP.

> Her concern would be to make sure that it remains a non-commercial activity to ensure fans are not exploited, and it is not being published, in the strict sense of traditional print publishing. - 2004 article

I had thought it should be fine since I am not charging any money and simply giving it away for free. I would love a source on the 25 copies and will likely contact her agent to see if they are ok with this.

If I talk about AI safety and the value this could bring to kids here, I believe it should be possible to get a green light, at least worth trying.


Re: people with high potential, yea so that is why I was targeting places like FOSS (Free and Open Source Software) conferences where youngsters who have traits like altruism, agency, intelligence, gather. I also went to the strategy board game events in the city where a diverse group of people who spend money and their free time on weekends to play complex board games with others.

Picking such locations help find people who have traits that are predictive of being likely to be nerd sniped into AI safety technical work. The other plan is to go to schools and hold competitions and give these away as prizes.

Damn it sounds like a great success in Russia, I would love to chat with you more about that, was there any follow up on these kids to see if reading it had any positive outcomes, even if it is self reported?

I am studying in India's top technical university and can talk to top rankers and Olypiad winners, the network effects seems doable since I have access to these people but I am not sure if I am ready to invest that much of my time and effort on this side project. I might reconsider after hearing how the russian experiment concluded.

Yeah I also did not read the book in a physical copy, I prefer reading it on my eink reader. I have found sending links to hpmor.com or just an epub is often not turning into real engagement with most people. These people often have a lot of pending bookmarked articles to read, videos to watch.

I am open to ideas for how better to package the online experience of this book or others. Maybe a review competition where the best 10 reviews of HPMOR gets a prize? I really don't want to force people too much to read. I just want people who would enjoy it if they read it casually to know that this book, and then the whole rationality rabbithole exists.