The Forensic Metascience Agent

The Forensic Metascience Agent is an AI agent equipped with 30+ tools for "sanity checking" the statistics and data presented in scientific papers. Much of it is based directly off of James Heathers' book An Introduction to Forensic Metascience.

The Forensic Metascience Agent is meant to compliment other automated and semi-automated systems for detecting improper image duplication and image manipulation like Proofig, ImageTwin, and RevewerZero.ai. It is also complementary to the copypaste detection system for scrutinizing open source datasets developed by Markus Englund at ScienceDetective.org.

Reported statistics are internally redundant: a mean of integer-scaled responses can only take certain values for a given sample size; a standard deviation is bounded by the range of the measurement; a t-statistic, its degrees of freedom, and its p-value must agree with one another; two "independent" studies should not report identical means and standard deviations down to the second decimal place. When the reported numbers violate these constraints, something is wrong — a typo, a copy-paste slip, a miscalculation, or worse.

What should be reported, and how?

At some point we might report every mathematical impossibility and statistical error we find on PubPeer. However, most errors are simple mistakes, and many are utterly inconsequential. The spectre of false positives is also very real when it comes to using AI, which is not 100% accurate at applying these tools (it may, for instance, apply GRIM on data that is actually continuous, not discrete). So, every finding found by the AI undergoes human review and scrutiny. Right now, it takes a lot for something to rise to the level of "we should report this on PubPeer / widely publicize this".

The toolkit

The 31 tools fall into the following families. Each check needs only the numbers printed in the paper — no raw data.

PDF & table ingestion

Statistics are read directly from the typeset PDFs — tables survive far better there than in text conversions — and normalized into structured cells before any check runs.

PDF extraction

Extracts text and tables from the main or supplementary PDF, page by page, so statistics can be read from the typeset tables rather than a lossy text conversion.

Inputs: PDF file, optional page range

Table parser

Normalizes a table — markdown, extracted PDF grid, or raw text — into structured cells (row label, column label, value, SD), and can reconcile two extractions of the same table, flagging cells that disagree.

Inputs: a table in any format; optional second extraction to cross-check

GRIM family

Integer data — Likert scales, counts, whole-number responses — can only produce certain means and percentages at a given sample size. These granularity tests check whether reported values are achievable at all.

GRIM

Tests whether a reported mean is possible for integer-scaled data: some integer sum must exist that rounds to the reported mean at the stated sample size and decimal precision.

Inputs: mean, sample size, scale bounds, decimal places

GRIM for percentages

The same granularity test applied to percentages, which carry more precision than means and therefore make the test considerably more powerful.

Inputs: percentage, sample size, decimal places

GRIM sweep

Given several percentages that supposedly share a denominator, finds every sample size at which all of them are simultaneously achievable. An empty result means the percentages cannot come from the same sample.

Inputs: list of percentages, maximum plausible sample size

2×2 reconstruction

Recovers the hidden cell counts of a 2×2 contingency table from two column percentages and the total N — optionally checking the reconstructed table against a reported χ².

Inputs: two column percentages, total N, optional reported χ²

GRIMMER & DEBIT

Standard deviations are granular too: for integer or binary data, only certain SDs can exist behind a given mean and sample size.

GRIMMER

Tests whether a reported SD is possible for integer data by enumerating the achievable sums and sums-of-squares (including a parity constraint between them) behind the reported mean.

Inputs: mean, SD, sample size, scale bounds

DEBIT

For binary (0/1) variables the SD is fully determined by the mean and sample size — this checks the reported SD against the only value it can be.

Inputs: mean, SD, sample size

DEBIT batch

Runs the DEBIT consistency check across every binary variable in a table at once, so entire demographic tables can be screened in one pass.

Inputs: a table of binary-variable means, SDs, and ns

Test-statistic recalculation

A test statistic, its degrees of freedom, and its p-value are redundant: they can be recomputed from the group summaries, and the recomputed values must agree with what the paper reports.

Independent-samples t-test

Recomputes t and p from two groups' means, SDs, and sizes (Student or Welch) and compares them with the reported values.

Inputs: each group's mean, SD, and n; reported t and/or p

Within-subjects t-test

For paired designs, recovers the pre/post correlation implied by the summary statistics and the reported result. An implied correlation greater than 1 in magnitude is mathematically impossible (Jané et al. 2024).

Inputs: pre/post means and SDs, sample size, reported t or p

One-way ANOVA

Recomputes the F statistic and p-value of a one-way ANOVA from the group means, SDs, and sizes.

Inputs: group means, SDs, and ns; reported F and/or p

F → p

Verifies that any reported F statistic and its degrees of freedom yield the reported p-value — usable for ANOVAs of any shape.

Inputs: F, both degrees of freedom, reported p

Chi-squared

Recomputes χ² and p from a contingency table's counts, with the Yates correction and Fisher's exact test for 2×2 tables.

Inputs: contingency-table counts; reported χ² and/or p

Regression coefficient

Verifies that a reported regression coefficient's t statistic equals B / SE and that its p-value matches the degrees of freedom.

Inputs: coefficient, standard error, degrees of freedom; reported t/p

Dispersion plausibility

The spread of a sample is bounded by its range, and simple bookkeeping identities must hold. Violations are outright impossible, not merely unlikely.

SD range check

The largest SD a sample can have is fixed by its range — approximately (range/2)·√(n/(n−1)). A reported SD above that maximum is impossible.

Inputs: SD, minimum and maximum possible values, sample size

SD-or-SE adjudication

When it is ambiguous whether a table reports standard deviations or standard errors, tests both readings (SE = SD/√n) against a plausibility ceiling and adjudicates which the value must be.

Inputs: reported value, sample size, a plausibility ceiling or range

Summation check

Verifies that values sum to their stated total — subgroup counts to N, percentages to 100 — within rounding tolerance.

Inputs: list of values, expected total, tolerance

Hidden information

Papers often report less than their own numbers reveal. These tools recover what a thresholded p-value or a bare summary conceals.

STALT

When a paper reports only “p < 0.05” but the group statistics allow exact recalculation, flags exact p-values many orders of magnitude below the stated threshold — information the paper concealed.

Inputs: recalculated exact p, reported threshold

SPRITE

Reconstructs the possible integer datasets behind a reported mean, SD, and sample size, surfacing distributions with implausible shapes — bimodal, or stacked at the scale boundaries — that suggest the summary statistics were invented.

Inputs: mean, SD, sample size, scale bounds

RCT baseline integrity

In a properly randomized trial, baseline characteristics should differ somewhat between groups by chance. Groups that are too similar are the classic signature of failed — or faked — randomization: a Table 1 where every baseline p-value is 0.93, 0.99, 0.99 is a red flag, not reassurance.

Carlisle–Stouffer–Fisher test

Combines a trial's baseline p-values into a one-sided test of excess similarity. A very small result means the groups are more alike than randomization can plausibly produce.

Inputs: the trial's baseline p-values

Bayesian Table-1 dispersion

Estimates a precision multiplier for the whole baseline table (in the style of Bolland 2022). Values well above 1 indicate under-dispersion: baseline differences systematically smaller than chance allows.

Inputs: every baseline row — means, SDs, and ns, or categorical counts

Proportion-from-normal

Tests whether a reported categorical proportion (e.g. “40% of participants above threshold X”) is consistent with the reported mean and SD of the underlying continuous variable.

Inputs: proportion, mean, SD, sample size, threshold and direction

Hand-calculation & heuristics

Some anomalies are not impossible numbers but improbable patterns: statistics computed backwards from a table's rounded values, or sample sizes too round to be real recruitment.

RIVETS (t-test)

Simulates data consistent with the printed, rounded summary statistics. A reported t that exactly equals the value hand-calculated from the rounded inputs — which almost no real underlying data reproduces — is the signature of statistics computed from the table rather than from data.

Inputs: group summaries; reported t and p

RIVETS (ANOVA)

The same hand-calculation detector applied to reported F statistics.

Inputs: group summaries; reported F and p

Round-n flag

Flags suspiciously round sample sizes (100, 200, 500…), which appear in fabricated studies far more often than real recruitment produces them.

Inputs: the study's sample sizes

Uninformative-statistic flag

Flags comparisons reported too coarsely — no test statistic, no exact p, no degrees of freedom — to pin down what was actually computed.

Inputs: whatever summary statistics the paper does report

Rank-sum granularity

The Mann–Whitney U statistic takes only (half-)integer values between 0 and n₁·n₂, so for given group sizes only a finite set of p-values is achievable.

GRIM-U

Checks whether a p-value attributed to a Mann–Whitney / Wilcoxon rank-sum test is achievable at the stated group sizes. A p-value below the U = 0 floor cannot be produced by any data and is a reporting error.

Inputs: both group sizes, reported p (exact or thresholded)

GRIM-U coexistence

Tests whether two nearly identical rank-sum p-values (say 0.171 and 0.172) can both be produced at the same group sizes, given the granularity of the achievable p-values.

Inputs: group sizes and the two reported p-values

Cross-table integrity

Two records of the same facts should be identical; identical records of different facts should not exist.

Duplication detection

Compares every (mean, SD) pair within and across a paper's tables and studies. Identical pairs recurring across supposedly independent samples are the signature of copy-pasted or recycled data — a place where duplication checks reach data that GRIM-style granularity tests cannot.

Inputs: labeled blocks of (mean, SD) cells, with sample identity per block

External tools

Established third-party checkers, run as part of the sweep.

statcheck

Runs the well-known statcheck tool: extracts APA-formatted test statistics from the text and recomputes their p-values, reporting inconsistencies and decision errors (where the recomputed p changes the significance conclusion).

Inputs: the paper's text or PDF