Nature’s 2016 Reproducibility Survey

In 2016, Nature surveyed 1,576 researchers about reproducibility in science for the feature “1,500 scientists lift the lid on reproducibility” (Baker, M. Nature 533, 452–454). The charts below are recomputed directly from the raw survey data, which is licensed under CC BY 4.0. Here we broke down the data further by discipline that was originally reported in the original article (we split "Physics and Engineering" into physics, engineering, and materials science, and we split out "psychology and social sciences" from "Other")

1,576

researchers surveyed

90%

say there is a significant or slight reproducibility crisis

72%

have tried and failed to reproduce another scientist's experiment

30%

have published a replication attempt (successful or failed)

Is there a reproducibility crisis? #

Which statement about a 'crisis of reproducibility' do you agree with? Each bar shows the response distribution and adds to 100%.

Fields use the same groups as the publishing breakdown below, including Physics and engineering combined and Other split into Psychology & social sciences and Other. Overall includes all 1,576 respondents.

Have you failed to reproduce an experiment? #

Share answering yes, by discipline — most scientists have experienced failure to reproduce results

In the experiment-failure and reproducibility-estimate charts, Physics & astronomy combines physics with astronomy and planetary science; Engineering and Materials science are shown separately. Psychology & social sciences groups write-in responses for psychology, cognitive science, economics, sociology, anthropology, political science, linguistics, communication, education and related fields from the survey’s original “Other” category. Mixed responses naming these fields are included; neuroscience-only responses remain in Other.

Have you established procedures for reproducibility? #

Share of respondents in each field, by when procedures were established. Each bar adds to 100%.

Overall includes all 1,576 respondents. Fields use the same groups as the crisis and publishing breakdowns.

Publishing replication attempts #

Have you ever tried to publish a reproduction attempt? Share of all 1,576 respondents

29.9%

of researchers (471 of 1,576) have published a replication attempt of some kind.

Publishing replication attempts by field #

Share of respondents in each field, ordered from highest to lowest published percentage in each panel. Successful and unsuccessful refer to the replication outcome.

Successful replication

Unsuccessful replication

Fields follow Nature’s original chart groups, including a combined Physics and engineering group. Other is split using the write-in areas of interest. Each percentage uses all respondents in that field; a respondent can report more than one publication experience.

How much published work in your field is reproducible? #

Distribution of answers within each discipline. Each panel shows the share of that discipline's respondents giving each estimate.

Chemistry

n = 106 · median estimate 70%

Physics & astronomy

n = 97 · median estimate 80%

Engineering

n = 66 · median estimate 60%

Materials science

n = 73 · median estimate 70%

Earth and environment

n = 95 · median estimate 70%

Biology

n = 703 · median estimate 60%

Medicine

n = 203 · median estimate 50%

Psychology & social sciences

n = 90 · median estimate 50%

Other

n = 143 · median estimate 60%

What factors contribute to irreproducible research? #

Share of all respondents rating each factor's contribution to failures to reproduce

What would improve reproducibility? #

Share of all respondents rating how likely each approach is to improve reproducibility

Source: Baker, M. “1,500 scientists lift the lid on reproducibility.” Nature 533, 452–454 (2016). doi:10.1038/533452a. All figures on this page are computed from the raw survey data released alongside the article.

Caveat: the survey was e-mailed to Nature readers and advertised on affiliated websites and social media as being about reproducibility, so the sample is self-selected and likely over-represents researchers already concerned with the issue. For yes/no questions, “I can’t remember” and blank answers count as no.