The first major project of The Metascience Observatory is to build a public database of experimental replications across all of science. There have already been several great studies assessing reproducibility, most famously in psychology and cancer biology. These were large, concerted efforts requiring large teams and lots of funding.

However, there are also one-off "garden variety" replication attempts. They are rare (perhaps 1% of papers), but there are thousands out there. Using both manual curation and AI/LLMs, we are creating a database of replications and compiling statistics on how reproducibility varies across fields, including fields like materials science and engineering, where insights into reproducibility are scarce.

A database of replications will enable new avenues of metascience research looking at "correlates of reproducibility". Additionally, we will be able to do p-curve analysis, z-curve analysis, and funnel plots at a larger scale than was possible before.

Longer term, we hope to create a reproducibility ranking for journals. Reproducibility rankings for journals could improve science by shifting focus away from impact factor (citation counts) towards the actual quality and rigor of scientific work. Previous work shows that citation counts are not correlated with reproducibility and that citation count is positively correlated with the chance that a paper will be retracted.