😌

Update 2

This update is in the format requested by the people evaluating the projects, otherwise it may overlap a lot with (and be directly copied from) the one below, apart from having more up to date data

1. How much money have you spent so far? Have you gotten more funding from other sources? Do you need more funding?

I have only spent about $50 on the domain name + analytics software, and I haven't gotten any funding from other sources. This is as expected as the funding was just to compensate me for doing it.

I'm not actively seeking any more funding at the moment, although I "wouldn't say no". I do have plans to continue this project (described in section 4). As with the original project this is something I plan to do in my own time, but would do a lot faster if I got some more funding or other encouragement.

2. How is the project going? (a few paragraphs)

Some links to the project:

• Announcement post here

• Actual site here

• Chrome extension here

Usage stats

For reference:

• This was written on 2023-08-30

• The website was released on 2023-08-10

• The Chrome extension was released on 2023-08-17

• I am using Datadog Real User Monitoring for the website analytics, and the Chrome developer dashboard for the extension analytics. Some limitations of this:

  • Datadog gets blocked by a lot of privacy settings, including using Brave as your browser

  • Sessions in the Chrome extension don't get counted in Datadog

  • I know the Chrome dashboard data is about 48 hours delayed

The stats:

• Since the announcement post, there have been 465 user sessions on the website. Some of these are me, so excluding all the UK ones to get a lower bound there are 403. This works out to just under 1 per hour. The average session duration is apparently 13.6 mins, which seems high to me. It's likely this is including a lot of idle time so maybe this isn't a very good metric but the session count seems reliable

• 16 people (excluding me) have placed bets via the calculator (i.e. entering their api key and clicking the "place bet" button). This surprisingly low to me, it's possible people prefer to just copy paste the amount into Manifold. The total amount bet (again excluding me) is M3450

• For the Chrome extension: there are 63 installs and 6 uninstalls in the developer dashboard, and around 20-30 daily users for the past week. It's worth noting I made a market on how many people would install it so these numbers may be inflated slightly (probably not though 😌)

• Aside from looking at the analytics, which may show people who try it once and think it's kind of cool but don't use it again, I know of about 5 people who clearly use it "a lot" and have messaged me about it on multiple occasions (and none are people I have personally coerced into using it)

Reflections

I'm happy with how the calculator itself turned out, I now use it myself almost every time I place a bet on Manifold. When I was working on it there were times when I thought it wouldn't be possible to get a nice clean version of it working because of the loan system, or because the api couldn't support what I was trying to do, so I'm glad those problems were overcome.

It's also true that getting this funding was counterfactual for actually finishing the project as it ended up taking much longer than expected. Without it I probably would have either given up or made a much more pared down version just for myself.

I'm pretty happy with the amount of usage it has received so far, especially since the market on the chrome extension took off. I'm also glad there weren't any major bugs and the feedback has been generally positive.

If I could do this over again the main thing I'd do differently would just be trying to get it out sooner. As you can see from the commit history, I basically put it on the back burner for a few months when I got busy with other stuff. There hasn't been much time to iterate on it or just let it spread by word of mouth and settle into people's workflows.

3. How well has your project gone compared to where you expected it to be at this point? (Score from 1-10, 10 = Better than expected)

I would say 8.5. I'm particularly happy that it is getting some significant usage, the big fear when putting out any project is that no-one will use it.

Secretly I hoped I would actually go way beyond the project description and have written a fully fledged portfolio management system by now. But, it's good enough to have something simpler that people use.

4. Are there any remaining ways you need help, besides more funding?

As alluded to above and below, I am planning to continue working on the successor to this project on the back burner, with the next steps roughly being:

• Write up some maths I did as part of this as a forum post or paper

• Make a more fully featured version separate to this calculator (with features summarised in this comment, the main one being "accounting for several estimates at the same time, and remembering previous bets")

• Make a version for Polymarket and/or other platforms

Anything that would help with these would be useful, including:

• Telling me you would use these things, which would make me do it faster

• More funding, which would also make me do it faster

• Sharing your relevant expertise and/or any pitching me on going in a certain direction

To any non-oracular-funders reading this comment, the places you can contact me where I am most likely to reply quickly are twitter and the EA Forum. Email also works.

Update

This is now done! Posting some links here:

• Announcement post here

• Actual site here

• Chrome extension here

This is about all I'm planning to do on this project before the 1st September deadline.

Some usage stats (written on 2023-08-21, project was released on on 2023-08-10). I'm using Datadog Real User Monitoring for analytics:

• Since the announcement post, there have been 302 user sessions. Some of these are me so excluding all the UK ones to get a lower bound there are 259. This works out to about 1 per hour.

• The average session duration is apparently 14.5 mins, which seems high to me. It's possible this is including a lot of idle time.

• 8 people (excluding me) have placed bets via the calculator (i.e. entering their api key and clicking the "place bet" button). This surprisingly low to me, it's possible people prefer to just copy paste the amount into manifold. The total amount bet (again excluding me) is around M1700.

• For the chrome extension, it was only published on 2023-08-17 and the data in the chrome developer dashboard seems to be a bit stale/unreliable (at one point it said it had more uninstalls that installs), but: the dashboard currently shows 6 installs and 4 uninstalls

General reflections on this project:

I'm happy with how the calculator itself turned out, I now use it myself almost every time I place a bet on Manifold. When I was working on it there were times when I thought it wouldn't be possible to get a nice clean version of it working because of the loan system, or because the api couldn't support what I was trying to do, so I'm glad those problems were overcome.

It's also true that getting this funding was counterfactual for actually finishing the project as it ended up taking much longer than expected. Without it I probably would have either given up or made a much more pared down version just for myself.

I'm reasonably happy with the amount of usage it has received so far, and the feedback it has gotten. Given the size of the manifold user base, at least a few percent of engaged Manifold users have used it a bit, and there are a couple of people who use it quite a lot as part of their main workflow for using Manifold.

If I could do this over again the main thing I'd do differently would just be trying to get it out sooner. As you can see from the commit history, I basically put it on the back burner for a few months when I got busy with other stuff. There hasn't been much time to iterate on it or just let it spread by word of mouth and settle into people's workflows.

My future plans for this are basically:

• Make a more fully featured version separate to this calculator (with features summarised in this comment)

• Make a version for Polymarket and/or other platforms

• ???

• Profit

(oh dear the editor killed some of my formatting, I hope you can still follow the comment)

Responding to comments:
@NathanYoung
> Surely I'd need to put all markets and all probs rather than just one?

Yes you're right, to do it completely correctly you would need to pull your whole portfolio, along with the probabilities you assign, and rebalance the whole thing every time you want to bet on a new market. This is the thing I started working on, and then I decided it was too complicated and to do something much simpler to begin with. The thing I am going to build initially is a calculator which makes some reasonable assumptions to come up with a pretty good answer (which is much better than doing it purely based on vibes).

There is an empirical finding about optimal betting in the case where you have multiple simultaneous markets to bet on, which is that the correct fraction to bet ends up being pretty proportional to what the dumb Kelly formula tells you. From Withrow (2007):

> When the number of bets is small, the optimal sizes of bet seem to be almost exactly proportional to the Kelly stakes on individual bets. This appears to be true for all such sets of data that the author has so far examined although only one example is presented here. A proof of this hypothesis should perhaps be attempted as part of future work. At present, it is left as an interesting but tentative observation. As the number of bets increases and the sum of stakes approaches 100% of the bankroll, the optimal bet sizes are somewhat different, although roughly monotonically related to the Kelly stakes. In fact, a better approximation to the optimal size of bet for a given event in this case is to use the edge 𝑝−𝑝̃ of that bet alone. These intriguing relationships were obtained empirically and it is unclear yet whether these or other relationships may be deduced rigorously (p. 622).

Informed by this, the thing I do by hand when I want to bet on a market is roughly:

- Come up with my estimate of the true probability

- Calculate the amount to bet based on the standard Kelly formula (which is valid when you only have 1 bet available, with fixed odds), scale this down by about half to be conservative

- Check if the expected rate of return this gives me is way worse than other markets I have already bet on, don't bet if so

- If the amount I have come up with is way more than the market can absorb, bet the market down to roughly my probability

- If this is less than the market can absorb, fiddle around with the numbers in the Kelly formula to account for market liquidity, then bet this amount, scaled up or down slightly based on vibes

And this is basically what I want to replicate as a simple calculator. The hard part of this project is reasoning through exactly what assumptions need to go into this, and how effective this can be expected to be compared to the full "portfolio rebalancing" solution. But I expect, based on the above paper and general experience, that the answer will be that you can get surprisingly far without accounting explicitly for every market. Building the actual website will then be pretty straightforward.


@Austin

> What tech stack are you building this in?

I'm going to use Next.js, and try and keep everything in the frontend for the initial version. If the maths gets too heavy I will add a django backend but I would like to avoid that for the first iteration.

> One concern I've always had with Kelly is that it doesn't seem to incorporate degree of certainty, making it seem hard to use in real contexts -- e.g. if two equally liquid markets are both at 20% and I think they should both be 50%, Kelly recommends the same course of action even if one is "Will this coin come up heads" and the other is "Will the president be republican in 2025". Does this seem true/like an issue to you?

I think the correct way to account for meta uncertainty like this in general is to defer to the market to some degree, and this turns out to imply fractional Kelly betting.

The argument for meta uncertainty being equivalent to deferring to the market:

If you see a market at 20% where you have literally no inside view on what the true probability is, you don't bet on it. This is the same as saying that you are fully deferring to the market for your estimate of the true probability, i.e. you're not just taking a uniform prior, because that would imply an expected probability of 50%. In order for not betting to make sense you have to believe something close to "if I bet $1 in either direction on this market, my expected profit is ~$0", which only makes sense if you think the market has the best estimate.

If you see a market at 20% where you do have a strong inside view (e.g. the coin flip one), you bet in such a way that maximises your utility, which means betting according to the Kelly formula for most people (who have something resembling logarithmic utility).

And then there is the majority of markets (like the Biden one), where you come up with your inside view estimate of 50%, but you feel it's much "more uncertain" than the coin flip case so you bet a lower amount. If you are still trying to maximise your expected utility then there exists some probability p_implied which is somewhere between the market's estimate and your inside view estimate, you can find this by inverting the Kelly forumula. So it's implied that you are putting some probability weight on "the market being right" and some on you being right. This is consistent with the two extreme cases, where you bet 0 because you completely defer to the market, or you bet exactly the Kelly amount because you completely trust your inside view.

The argument for this implying fractional Kelly betting:

The paper I linked to has a good justification of this (https://people.cs.umass.edu/~wallach/workshops/nips2011css/papers/Beygelzimer.pdf).
TL;DR:

The Kelly formula is (p - p_market) / (1 - p_market), and in practice people often scale this down by some factor lambda < 1. This is exactly equivalent to putting a probability weight of lambda on you being right, and a weight of (1 - lambda) on the market being right.

This does then leave the problem of "user education". i.e. if people don't defer to the market in any way they will find that Kelly betting makes them overconfident, but this is not a deep problem with Kelly betting itself