Previous replication database initiatives

Curate Science

This was a website documenting and summarizing the individual efforts as well as large-scale collective projects in the social sciences. Some of the information there has been ingested into our replication database. It can be viewed on the Internet Archive.

PsychFileDrawer.org

This was an online tool designed to archive replication reports in psychology. We attempted scraping this data but found it difficult to work with and many of the studies were already in our dataset. This resource went offline a few years ago but can be viewed on the Internet Archive.

Economics Replication Wiki

This was a website created by Jan H. Hoeffler that compiled published replications of empirical studies in economics. The wiki operated with papers being the unit of replication, not experiments or individual effects. We found that it had 791 "replication studies" before being taken down. Many were not categorized regarding the outcome, but when categorized they typically used a binary label ("successful” or “failed”). We scraped the data, but found it difficult to work with. The "replications" were usually not direct replications of experiments, and were more along the line of conceptual replications, but it was difficult to determine which studies should be cited as the originals. At other times, papers were actually technical replications (for instance replicating a forecasting model by simply reanalyzing the same data using the same code provided by the original authors). Other times, what was considered a "replication" used the same exact data as the original study to study the same hypothesis but analyzed the data differently using what the authors thought was a better technique. The website offered this categorization scheme (i) same data, same code, (ii) new data, same methods, (iii) same data, new methods, (iv) new methods, new data. However, most papers were not categorized. The Metascience Observatory scraped studies that had been categorized as (ii), (iii), (iv), or had been marked as "close" replications. We processed about 300 paper with our AI pipeline, about 260 of which ended up in our database. The replication wiki can be viewed on the Internet Archive.