Project summary
Public AI leaderboards report ranks without saying which ranks are actually distinguishable. I have already measured this with the boards' own published per-item results on eleven boards from five fields (SWE-bench Verified/Lite/test, HELM, MTEB, LiveBench, MathArena, LMArena, ProteinGym, TabArena, CASP14). On SWE-bench Verified, 129 of 133 adjacent rank pairs are not statistically separable and the top 8 systems cannot be told apart from #1 (exact McNemar on disagreeing instances; parity with the official board 133/133). On HELM, 89 of 89 adjacent pairs are not separable. LMArena, which ranks on paired human votes, is the positive control where ranks do separate. A separate test on the HuggingFace Open LLM Leaderboard showed that a model's v1 rank predicted its drop when the board moved to decontaminated v2 (Spearman rho = 0.21, p = 2e-5, n = 397), i.e. part of the v1 rank was fit to v1's own benchmark mix. The SWE-bench result was independently replicated by a third party with their own code (SWE-bench/experiments issue #466, Aug 2026), who also showed that under Holm correction 0 of 133 adjacent pairs separate and that the minimum detectable difference on that board is 5.4 percentage points. The five-benchmark paper is on Zenodo (10.5281/zenodo.22541065, v4, Sept 2026); everything is pre-registered, open-source and reproducible from public files.
This grant turns those one-off audits into a maintained tool and a standing public report: a resolution column (rank confidence interval, Holm-corrected separability, minimum detectable difference, and an "indistinguishable from #1" flag) for every system on 5-6 major leaderboards, re-run every quarter, with the pre-registration and a verification bundle published each time.
What are this project's goals? How will you achieve them?
Goals over 6 months:
1. Generalise the tool. One script that takes any leaderboard's per-item results and produces rank confidence intervals, adjacent-pair separability with Holm family-wise correction, a minimum-detectable-difference column, and a "top cluster" size, with self-checks that fail the run (identity p = 1; planted flips detected; Wilson interval matches a published value). The per-board scripts exist for eleven boards; this turns them into one maintained tool with a common input format. Boards are added only where per-item results are public (checked Nov 2026: LiveCodeBench publishes per-problem eval files per model; ARC Prize publishes per-task scores for verified runs; BigCodeBench publishes generations but not per-task pass/fail, so it would need re-execution). Boards are chosen by availability of per-item outputs, not by expected result.
2. Quarterly public report, each one pre-registered before the data pull: which boards changed resolution, which "new #1" claims are inside the noise, which boards got better or worse. Published on Zenodo with a DOI and a verification capsule (sha256 manifest plus an offline verify script).
3. Offer the resolution column upstream to each board as a pull request. Where maintainers decline, the report still exists.
Method: paired per-item comparison (exact McNemar where items are shared), bootstrap rank intervals, and a parity check against the official board before any number is reported. Pre-registration is locked by commit hash before each run. Negative or null results are published the same way as positive ones.
How will you know it worked?
Measurable targets, all checkable from public artefacts:
Two quarterly reports published on Zenodo within the 6 months, each with a pre-registration commit dated before its data pull and a verification capsule that re-runs offline.
- The tool runs end to end on at least 5 boards with the parity gate passing (recomputed rates match the official board for every entry, or the mismatch is printed and the entry is left blank).
- At least one more independent replication or correction from someone who is not me, on the public repo or the reports. The August 2026 replication on SWE-bench is the template.
- Stretch, not required: a board adopts the resolution column, or a board's maintainers cite the report when changing their presentation.
If after two quarters the reports show nothing changing, the project has still worked: the result is a dated, verifiable statement of how much ranking each board supports.
How will this funding be used?
About $5,500: my working time, roughly 2 days per week for 6 months. I work part-time in logistics; this buys the hours.
- About $1,500: compute and API costs for the AI assistant that writes and runs the code, and for re-running full-board bootstraps.
- About $1,000: Zenodo and verification tooling, and a reserve for a second re-run if a board changes its format mid-quarter.
If only the minimum ($3,000) is reached: the tool generalisation and one quarterly report, on 3 boards instead of 5-6.
Who is on your team? What's your track record on similar projects?
One person: Ilpo Väätäinen, Helsinki, independent, no institution. I am not a domain expert. I direct an AI assistant that writes the code, runs the analyses and drafts the text, including this proposal; I choose the questions, lock the pre-registrations, check the outputs against the sources and decide what gets published. Everything below was made this way and says so.
Track record on this exact topic:
"How Much of a Leaderboard Ranking Survives Its Own Sampling Error? A Paired-Resolution Audit of Five AI Benchmarks" (Zenodo 10.5281/zenodo.22541065, v4, 19 Sept 2026; code and matrices at github.com/ipezygj/swebench-rank-audit, eleven boards). Issue #466 on SWE-bench/experiments proposes the resolution column; an independent replication with different code was posted there in Aug 2026. PR #60 on swe-bench.github.io implements the column at site build time.
- HELM extension (Aug 2026): 89/89 adjacent pairs not separable, exact parity with the published mean win rate.
- HuggingFace Open LLM Leaderboard v1 to v2 contamination test (Oct 2026): pre-registered, result in the summary.
- "The Benchmark Number Is Not the Deployment Number" (Zenodo 10.5281/zenodo.22109472): rank collapse and base-rate collapse when a benchmark score is read as a deployment number.
- Wider practice: around 14 Zenodo-deposited pre-registered audits across fields (astrophysics, seismology, public statistics), several in journal review. The common method is the same: lock the test first, run against the real sample, publish either outcome.
What are the most likely causes and outcomes if this project fails?
Per-item results stop being public. Some boards publish only aggregates, and LiveCodeBench archived its submissions repo in Oct 2026. Then the tool covers fewer boards; the report says which boards cannot be audited, which is itself a finding.
- Maintainers ignore the pull requests. Likely: my two SWE-bench PRs have had no maintainer response since August. The report still gets published; adoption is a bonus, not the deliverable.
- The result is boring, i.e. boards are already well resolved. Then the quarterly report says so with numbers. That is the honest outcome and it is cheap.
- The failure that would matter: a parity mismatch I do not catch, so a reported "indistinguishable" is wrong. Mitigation: parity against the official board is a hard gate before any number is written, and every run ships the verification capsule so anyone can re-check offline.
How much money have you raised in the last 12 months, and from where?
$0 raised. Three applications are pending for related but non-identical work (Finnish Cultural Foundation, Foresight AI for Science & Safety, EA Transformative AI Fund); none of them funds this leaderboard tool specifically. If any of them is awarded, I will say so on this page and reduce the ask accordingly.