The first major project of The Metascience Observatory is to build a publicly accessible database of experimental replications across all of science.

There have already been many great studies assessing reproducibility, such as the tour de force studies in psychology and cancer biology spearheaded by Brian Nosek, the "Many Labs" initiatives in psychology, and the Transparent Replications project run by Spencer Greenberg’s team at clearerthinking.org. These were large, concerted efforts requiring many hours of work and lots of funding.

However, there are also one-off “garden variety” replication attempts. They are rare (perhaps 1% of papers), but there are thousands out there. Using both manual curation and AI/LLMs, we will create a database of replications and statistics on how reproducibility varies across fields. Unlike the FORRT replication database, which covers only psychology, we are covering all of science, including fields like materials science and engineering where there is not a lot of hard data on reproducibility.

A database of replications will enable many interesting avenues of metascience research looking at "correlates of reproducibility". The database will also help people quickly determine if a finding has replicated. After all, it is only after independent experiments consistently demonstrate a claim that we can really start to believe it strongly. (This is true regardless of whether you are a verificationist, falsificationist, Bayesian, Popperian, etc.)

Longer term, we may be able to create a reproducibility ranking for different journals. Reproducibility rankings for journals could improve science by shifting focus away from impact factor (citation counts) towards the actual quality and rigor of scientific work. Previous work shows that citation numbers are not correlated with reproducibility and that citation rates are positively correlated with the chance that a paper will be retracted.

The Metascience Observatory is currently fiscally sponsored by Mind First Foundation, a 501c3 nonprofit based in Massachusetts. In the future, we will likely spin out as a separate nonprofit initiative. We are still formalizing our advisory board, but so far we have received informal advice from James Heathers, Paul Litvak, and Alex Holcombe.

Longer term plans (beyond our minimal funding level)

Money received beyond our initial $25,000 goal will help The Metascience Observatory cover API costs to scale up our replications database further and hone our AI paper analysis pipeline. Additionally, The Metascience Observatory has longer term plans which are contingent on additional funding beyond our minimum funding level:

• Using our database to train an AI system to predict how likely a paper is to replicate.

• Doing p-curve, z-curve, and funnel plot analyses across different fields.

• Articles explaining our approach to metascience.

• New approaches to automated meta-analysis.

• New approaches to detecting anomalous data / fraud.

Why am I (Dan Elton) qualified to work on this?

I am quite familiar with academic research, as I was a PhD student in physics for six years, followed by two years as a postdoc and three years as a Staff Scientist at NIH. I am a co-author on about 45 peer-reviewed papers in physics, AI for molecular discovery, and AI for medical imaging. I've written eight articles about metascience topics on my Substack. The most notable is "A Defense of Peer Review", an article I did for Asimov Press which required over 100 hours of research.