Other replication database initiatives

Ongoing initiatives

FORRT Replication Database (FReD)

The FORRT Replication Database (FReD) is a crowdsourced database of replication attempts, primarily in psychology, maintained by the Framework for Open and Reproducible Research Training (FORRT) and led by Lukas Röseler and colleagues. It can be browsed through the FReD Explorer web app, and is also available as a free and open-source R package. FReD is a major source for our own replications database: a large share of our rows were ingested from FReD snapshots.

FLoRA Explorer

The FLoRA Explorer presents FLoRA (the FORRT Library of Replication Attempts), a newer FORRT effort that builds on FReD and tracks "repetition attempts" across the sciences and humanities. It distinguishes three types of repetition: numerical reproductions, robustness reproductions, and replications. The explorer lets you browse and filter studies by discipline, year, and journal, and includes analyses of citation impact and of authorship overlap between original and replication teams, along with integrations such as a Zotero plugin and the Replication Atlas. Unlike FReD (and unlike our database), FLoRA does not gather statistical data such as effect sizes, sample sizes, or p-values — each entry records only a standardized outcome, based on how the authors of the repetition report characterized their result ("successful", "failed", or "mixed" for replications; computational success and robustness codes for reproductions). Entries are recorded at the level of the repetition attempt rather than the individual finding: each row links one repetition report to the original study it targeted and carries a single overall outcome, so a replication that tested several findings from a paper appears as one entry. (Our database, by contrast, records one row per replicated effect.)

OpenMKT — Replications of Marketing Studies

OpenMKT maintains a catalog of high-powered, preregistered direct replications of marketing studies, available for browsing and download as a CSV. As of its last update the catalog tracked 43 such replications, of which only 5 were unambiguously successful.

Past initiatives

Curate Science

This was a website documenting and summarizing the individual efforts as well as large-scale collective projects in the social sciences. Some of the information there has been ingested into our replication database. It can be viewed on the Internet Archive.

PsychFileDrawer.org

This was an online tool designed to archive replication reports in psychology. We attempted scraping this data but found it difficult to work with and many of the studies were already in our dataset. This resource went offline a few years ago but can be viewed on the Internet Archive.

Economics Replication Wiki

This was a website created by Jan H. Hoeffler that compiled published replications of empirical studies in economics. The wiki operated with papers being the unit of replication, not experiments or individual effects. We found that it had 791 "replication studies" before being taken down. Many were not categorized regarding the outcome, but when categorized they typically used a binary label ("successful” or “failed”). We scraped the data, but found it difficult to work with. The "replications" were usually not direct replications of experiments, and were more along the line of conceptual replications, but it was difficult to determine which studies should be cited as the originals. At other times, papers were actually technical replications (for instance replicating a forecasting model by simply reanalyzing the same data using the same code provided by the original authors). Other times, what was considered a "replication" used the same exact data as the original study to study the same hypothesis but analyzed the data differently using what the authors thought was a better technique. The website offered this categorization scheme (i) same data, same code, (ii) new data, same methods, (iii) same data, new methods, (iv) new methods, new data. However, most papers were not categorized. The Metascience Observatory scraped studies that had been categorized as (ii), (iii), (iv), or had been marked as "close" replications. We processed about 300 paper with our AI pipeline, about 260 of which ended up in our database. The replication wiki can be viewed on the Internet Archive.