The Forensic Metascience Agent

The Forensic Metascience Agent is an AI agent we are developing that reads a scientific paper — the body text, the tables in the main PDF, and any supplementary-information PDFs — and checks every checkable statistic for internal consistency. It applies a battery of 31 statistical forensics tools, ranging from classic granularity tests like GRIM to randomized-trial baseline checks in the tradition of John Carlisle's anesthesiology audits, and then writes an editorial review that ends in a plain-language trust verdict: trustworthy, questionable, or untrustworthy.

Why forensic metascience?

A surprising amount of error — and occasionally fabrication — can be detected from a published paper alone, without any access to the raw data. Reported statistics are internally redundant: a mean of integer-scaled responses can only take certain values for a given sample size; a standard deviation is bounded by the range of the measurement; a t-statistic, its degrees of freedom, and its p-value must agree with one another; two "independent" studies should not report identical means and standard deviations down to the second decimal place. When the reported numbers violate these constraints, something is wrong — a typo, a copy-paste slip, a miscalculation, or worse.

Checking these constraints by hand is tedious, error-prone work that a small community of specialists ("data thugs," as they've affectionately been called) has done for years. The methods themselves are well documented — our agent's toolkit implements the techniques catalogued in James Heathers' An Introduction to Forensic Metascience (2025) — but applying them exhaustively to a paper takes hours of expert attention. This is exactly the kind of systematic, rule-governed work that AI agents are well suited to: sweep every statistic, run the right check on each one, and record the results in a form a human can verify.

How it works

The agent runs a two-stage pipeline built on the Claude Agent SDK.

Stage 1 — the forensic sweep. An agent armed with all 31 forensic tools (exposed via an MCP server) reads the paper and works through it statistic by statistic: means and SDs are checked for granularity and range violations, test statistics are recomputed from group summaries, thresholded p-values are recalculated exactly, baseline tables in randomized trials are tested for excessive similarity, and all tables are cross-compared for duplicated data. The output is a schema-validated JSON file of structured findings. Every finding records the technique used, the exact inputs read from the paper, the computed versus reported values, a severity rating, and a reproduction command — so any human can re-run the exact check from the command line and confirm it.

Stage 2 — the editorial review. A second pass triages the findings and adds the contextual reasoning a numeric sweep can't: Is the headline effect size plausible against external benchmarks? Does the statistical analysis match the study design? Does the paper claim its data are available while actually withholding them? It then writes a narrative report, leading with the most important problem and ending with the trust verdict.

Every finding carries one of five severity levels:

impossible — violates mathematics (e.g., a GRIM failure, an SD above the maximum possible for the measurement range, an implied correlation greater than 1)
highly suspicious — technically possible but extremely unlikely under honest reporting
suspicious — unusual but plausible (e.g., a recalculated statistic disagreeing beyond rounding)
indeterminate — cannot be determined from the available information
consistent — passes the check

The toolkit

The 31 tools fall into the following families. Each check needs only the numbers printed in the paper — no raw data.

PDF & table ingestion

Statistics are read directly from the typeset PDFs — tables survive far better there than in text conversions — and normalized into structured cells before any check runs.

PDF extraction

Extracts text and tables from the main or supplementary PDF, page by page, so statistics can be read from the typeset tables rather than a lossy text conversion.

Inputs: PDF file, optional page range

Table parser

Normalizes a table — markdown, extracted PDF grid, or raw text — into structured cells (row label, column label, value, SD), and can reconcile two extractions of the same table, flagging cells that disagree.

Inputs: a table in any format; optional second extraction to cross-check

GRIM family

Integer data — Likert scales, counts, whole-number responses — can only produce certain means and percentages at a given sample size. These granularity tests check whether reported values are achievable at all.

GRIM

Tests whether a reported mean is possible for integer-scaled data: some integer sum must exist that rounds to the reported mean at the stated sample size and decimal precision.

Inputs: mean, sample size, scale bounds, decimal places

GRIM for percentages

The same granularity test applied to percentages, which carry more precision than means and therefore make the test considerably more powerful.

Inputs: percentage, sample size, decimal places

GRIM sweep

Given several percentages that supposedly share a denominator, finds every sample size at which all of them are simultaneously achievable. An empty result means the percentages cannot come from the same sample.

Inputs: list of percentages, maximum plausible sample size

2×2 reconstruction

Recovers the hidden cell counts of a 2×2 contingency table from two column percentages and the total N — optionally checking the reconstructed table against a reported χ².

Inputs: two column percentages, total N, optional reported χ²

GRIMMER & DEBIT

Standard deviations are granular too: for integer or binary data, only certain SDs can exist behind a given mean and sample size.

GRIMMER

Tests whether a reported SD is possible for integer data by enumerating the achievable sums and sums-of-squares (including a parity constraint between them) behind the reported mean.

Inputs: mean, SD, sample size, scale bounds

DEBIT

For binary (0/1) variables the SD is fully determined by the mean and sample size — this checks the reported SD against the only value it can be.

Inputs: mean, SD, sample size

DEBIT batch

Runs the DEBIT consistency check across every binary variable in a table at once, so entire demographic tables can be screened in one pass.

Inputs: a table of binary-variable means, SDs, and ns

Test-statistic recalculation

A test statistic, its degrees of freedom, and its p-value are redundant: they can be recomputed from the group summaries, and the recomputed values must agree with what the paper reports.

Independent-samples t-test

Recomputes t and p from two groups' means, SDs, and sizes (Student or Welch) and compares them with the reported values.

Inputs: each group's mean, SD, and n; reported t and/or p

Within-subjects t-test

For paired designs, recovers the pre/post correlation implied by the summary statistics and the reported result. An implied correlation greater than 1 in magnitude is mathematically impossible (Jané et al. 2024).

Inputs: pre/post means and SDs, sample size, reported t or p

One-way ANOVA

Recomputes the F statistic and p-value of a one-way ANOVA from the group means, SDs, and sizes.

Inputs: group means, SDs, and ns; reported F and/or p

F → p

Verifies that any reported F statistic and its degrees of freedom yield the reported p-value — usable for ANOVAs of any shape.

Inputs: F, both degrees of freedom, reported p

Chi-squared

Recomputes χ² and p from a contingency table's counts, with the Yates correction and Fisher's exact test for 2×2 tables.

Inputs: contingency-table counts; reported χ² and/or p

Regression coefficient

Verifies that a reported regression coefficient's t statistic equals B / SE and that its p-value matches the degrees of freedom.

Inputs: coefficient, standard error, degrees of freedom; reported t/p

Dispersion plausibility

The spread of a sample is bounded by its range, and simple bookkeeping identities must hold. Violations are outright impossible, not merely unlikely.

SD range check

The largest SD a sample can have is fixed by its range — approximately (range/2)·√(n/(n−1)). A reported SD above that maximum is impossible.

Inputs: SD, minimum and maximum possible values, sample size

SD-or-SE adjudication

When it is ambiguous whether a table reports standard deviations or standard errors, tests both readings (SE = SD/√n) against a plausibility ceiling and adjudicates which the value must be.

Inputs: reported value, sample size, a plausibility ceiling or range

Summation check

Verifies that values sum to their stated total — subgroup counts to N, percentages to 100 — within rounding tolerance.

Inputs: list of values, expected total, tolerance

Hidden information

Papers often report less than their own numbers reveal. These tools recover what a thresholded p-value or a bare summary conceals.

STALT

When a paper reports only “p < 0.05” but the group statistics allow exact recalculation, flags exact p-values many orders of magnitude below the stated threshold — information the paper concealed.

Inputs: recalculated exact p, reported threshold

SPRITE

Reconstructs the possible integer datasets behind a reported mean, SD, and sample size, surfacing distributions with implausible shapes — bimodal, or stacked at the scale boundaries — that suggest the summary statistics were invented.

Inputs: mean, SD, sample size, scale bounds

RCT baseline integrity

In a properly randomized trial, baseline characteristics should differ somewhat between groups by chance. Groups that are too similar are the classic signature of failed — or faked — randomization: a Table 1 where every baseline p-value is 0.93, 0.99, 0.99 is a red flag, not reassurance.

Carlisle–Stouffer–Fisher test

Combines a trial's baseline p-values into a one-sided test of excess similarity. A very small result means the groups are more alike than randomization can plausibly produce.

Inputs: the trial's baseline p-values

Bayesian Table-1 dispersion

Estimates a precision multiplier for the whole baseline table (in the style of Bolland 2022). Values well above 1 indicate under-dispersion: baseline differences systematically smaller than chance allows.

Inputs: every baseline row — means, SDs, and ns, or categorical counts

Proportion-from-normal

Tests whether a reported categorical proportion (e.g. “40% of participants above threshold X”) is consistent with the reported mean and SD of the underlying continuous variable.

Inputs: proportion, mean, SD, sample size, threshold and direction

Hand-calculation & heuristics

Some anomalies are not impossible numbers but improbable patterns: statistics computed backwards from a table's rounded values, or sample sizes too round to be real recruitment.

RIVETS (t-test)

Simulates data consistent with the printed, rounded summary statistics. A reported t that exactly equals the value hand-calculated from the rounded inputs — which almost no real underlying data reproduces — is the signature of statistics computed from the table rather than from data.

Inputs: group summaries; reported t and p

RIVETS (ANOVA)

The same hand-calculation detector applied to reported F statistics.

Inputs: group summaries; reported F and p

Round-n flag

Flags suspiciously round sample sizes (100, 200, 500…), which appear in fabricated studies far more often than real recruitment produces them.

Inputs: the study's sample sizes

Uninformative-statistic flag

Flags comparisons reported too coarsely — no test statistic, no exact p, no degrees of freedom — to pin down what was actually computed.

Inputs: whatever summary statistics the paper does report

Rank-sum granularity

The Mann–Whitney U statistic takes only (half-)integer values between 0 and n₁·n₂, so for given group sizes only a finite set of p-values is achievable.

GRIM-U

Checks whether a p-value attributed to a Mann–Whitney / Wilcoxon rank-sum test is achievable at the stated group sizes. A p-value below the U = 0 floor cannot be produced by any data and is a reporting error.

Inputs: both group sizes, reported p (exact or thresholded)

GRIM-U coexistence

Tests whether two nearly identical rank-sum p-values (say 0.171 and 0.172) can both be produced at the same group sizes, given the granularity of the achievable p-values.

Inputs: group sizes and the two reported p-values

Cross-table integrity

Two records of the same facts should be identical; identical records of different facts should not exist.

Duplication detection

Compares every (mean, SD) pair within and across a paper's tables and studies. Identical pairs recurring across supposedly independent samples are the signature of copy-pasted or recycled data — a place where duplication checks reach data that GRIM-style granularity tests cannot.

Inputs: labeled blocks of (mean, SD) cells, with sample identity per block

External tools

Established third-party checkers, run as part of the sweep.

statcheck

Runs the well-known statcheck tool: extracts APA-formatted test statistics from the text and recomputes their p-values, reporting inconsistencies and decision errors (where the recomputed p changes the significance conclusion).

Inputs: the paper's text or PDF

Guardrails

A tool that scans papers for anomalies must be careful about false accusations, so the agent operates under an explicit rulebook that encodes hard-won lessons from the forensic metascience community:

Impossibility beats implausibility. A single verified mathematical impossibility outranks any number of merely suspicious findings — but the inputs must be verified first. Misapplied GRIM (running it on continuous data like body weight) is the most common source of false accusations, and the agent is explicitly instructed against it.
No invented constraints. A finding that depends on an assumed-but-unstated bound (say, a guessed plausible range for adult height) is capped in severity and can never be the headline problem of a review.
Language discipline. Reports say "anomaly," "inconsistency," and "warrants clarification." Words like "fabrication" are reserved for verified impossibilities — and even then phrased as "consistent with" rather than as an accusation.
Reproducibility. Every finding includes its exact inputs and a command-line reproduction string, so nothing rests on the agent's say-so.

The agent is a triage and second-opinion tool, not an accusation engine. Its job is to surface things that warrant a closer human look — and to make that closer look easy.

Evaluation

We evaluate the pipeline against gold-standard fixtures hand-derived from in-depth reviews by expert human forensic reviewers. Automated scoring measures how many of the expert's key findings the agent recovered (weighted by importance), how many false positives it introduced, and whether its overall verdict agreed with the expert's. The statistical tools themselves are covered by a suite of roughly 190 unit tests. The agent does not yet match a careful human expert — in our evaluations it recovers many, but not all, of the findings an expert surfaces — and we treat every automated review as a draft for human verification rather than a final judgment.

Get in touch

We are actively developing this project and are interested in collaborating with researchers, journals, and institutions who want to apply forensic screening at scale. If you'd like to learn more, contact us.