AI-Powered Systematic Review Reproducibility Standards Are Lagging the Tools

Systematic reviews are sold on reproducibility. The PRISMA 2020 statement, the field's reporting standard, runs through 27 checklist items and says nothing about large language models, screening prioritization algorithms, or automated data extraction. A colleague I'll call Dr. R found the gap the hard way when a journal asked for the model-assigned relevance scores behind her 6,000-record screen. She had the final included list. The journal wanted the scores. She had deleted the project workspace after submission.

The tools are already in routine use. Covidence, Rayyan, EPPI-Reviewer, DistillerSR, and a growing set of in-house pipelines now offer some form of machine-assisted screening or extraction. Researchers use them to rank citations, flag likely excluded records, and pull study characteristics into structured tables. The difficulty is not that the tools fail. Reviewers often cannot show what the model did, and that is the reproducibility problem.

What the standards do and do not require

The PRISMA 2020 statement demands that reviewers describe the information sources, the search strategy, the screening process, and the data collection process with enough detail for someone else to repeat the review. If researchers used Rayyan's AI ranking or a large language model to extract effect sizes, a rigorous reading of item 9 and item 10 would require naming the tool, its version, the model in use, and the point at which a human overrode the machine. What journals actually request varies sharply, and few checklists have a dedicated AI disclosure field.

AMSTAR 2, the 16-item quality instrument for systematic reviews of healthcare interventions, asks about protocol registration, comprehensive search, and justified exclusions. It does not ask whether a model was retrained or whether prompts were saved. That absence matters because systematic reviews increasingly feed clinical guidelines, health technology assessments, and funding decisions. A review that cannot be reconstructed may still be useful; it is not fully reproducible in the sense the method promises.

When a model is in the loop, the audit trail changes

A traditional systematic review has a paper trail: search strings, exported RIS files, screening forms, extracted spreadsheets. An AI-assisted review has additional artifacts that are less stable. Model weights change. Prompts are edited at 11 p.m. before a meeting. Online platforms can update their underlying classifiers without the review team noticing. The PRISMA 2020 checklist assumes stable methods; reproducibility standards now have to assume the opposite.

In practice, a team should freeze four things before screening starts: the tool and its version, the model name and variant, the prompt or system instruction, and the decision rule for human reconciliation. Some platforms keep a record of relevance scores; many do not export them in a form tied to the deduplicated record ID. Dr. R's problem was not that she misused the tool. The journal's request was reasonable. She simply had no output artifact that survived cleanup.

This is not an argument against using the tools. The efficiency gains are real, but they vary by corpus size and by the amount of conflict resolution a team is willing to do. Shops that use machine triage only for title and abstract screening often report saving between a quarter and half of the human effort in that phase. That saving disappears if the team must re-run the screen later to satisfy a reproducibility query, which is exactly what happens to teams that do not preserve model outputs.

Photo by Nahrizul Kadri on Unsplash

The failure points are not where teams expect them

Screening prioritization rarely fails by misclassifying included studies; it fails by producing scores that cannot be interpreted later. Data extraction failures are more visible. A model can transpose a mean and standard deviation from a table in which the row order changes, or it can omit a comparator arm because the paper labels treatment groups by brand name in one paragraph and by generic name in another. Human reviewers make these errors too, but humans can explain the path that led to a wrong value when asked. A model cannot yet reconstruct the path that produced the wrong value.

This is why the override log matters more than the model score. The log turns the black box into a decision record: which records the model ranked high and the team excluded, which records the model ranked low and the team included, and the one-line reason for each reversal. Those reversals are the most informative part of any AI-assisted review. They show where the machine's judgment diverged from clinical or subject-matter judgment, and they are the first thing a reproducibility audit should examine.

What reproducible AI-assisted reviews would need to record

If journals are serious about reproducibility in evidence synthesis, the reporting framework needs to catch up. A minimal reproducibility bundle for an AI-assisted systematic review would include the protocol version, the exact search strings, the deduplicated record set, the screening decisions before and after machine ranking, the model version and prompt file, and the override log for any record where a human reversed the machine's classification.

Cochrane has signaled caution in methods guidance, generally treating automated tools as acceptable for initial prioritization while retaining independent human verification for final inclusion. The Cochrane Handbook remains the reference point for intervention reviews, and its approach is conservative: automation should reduce workload without removing the human check that systematic reviews rely on.

For data extraction, the stakes are higher. A large language model can misread a confidence interval, copy an arm label from the wrong row, or generate a plausible value that did not appear in the source. Journals and preprint servers are already dealing with AI-related integrity problems elsewhere in the publishing pipeline, including tools that flag doctored research data and updated policies on generative AI in peer review. Systematic review reporting has not yet produced an equivalent set of specific checks.

What this means for your lab

Most review teams I meet already use Covidence or Rayyan for screening; the gap is not adoption, it is record-keeping. The fix is not a policy document. It is a clean artifacts folder that contains the protocol, search strings, deduplicated record IDs, extracted data, and the model outputs at each stage. Treat the AI tool as you would a laboratory instrument: log the serial number, calibration settings, and any changes mid-experiment.

• Freeze the model version and prompt before the first screening session begins.
• Export machine-assigned scores with the record ID, not as a separate list that cannot be rejoined to the dataset.
• Keep a timestamped override log whenever a human reverses a machine inclusion or exclusion decision, with a one-line reason for each reversal.
• State in the methods section which tasks were machine-assisted, which model did them, and whether the human reviewer could see the machine label before making a decision.
• Store all files in a repository with a versioned snapshot at protocol registration, after screening, and after extraction.

A co-investigator once considered a 3,000-record clarification exercise after a journal query. The lab avoided it only because they had saved a screenshot of the tool's decision screen and an exported file of relevance scores tied to record IDs. That is the standard teams should aim for, not because journals ask often, but because the queries arrive after the short-term memory of the project has gone.

Photo by Markus Spiske on Unsplash

The next set of questions journals should ask

Editors and peer reviewers cannot evaluate what authors do not disclose. A short supplementary checklist would close much of the gap: name the AI tool, name the model, state the model version, state whether the tool performed screening, extraction, or both, and state whether a human independently verified every included study or only a sample. Journals that publish evidence syntheses could add these questions to their submission form without waiting for a formal revision to PRISMA.

Toolmakers also have a role. Platforms such as Covidence, Rayyan, EPPI-Reviewer, and DistillerSR already track some workflow decisions, but the exports are not standardized across platforms. A portable JSON export of screening decisions, model scores, and override events would solve much of Dr. R's problem. Until that exists, teams need to build the export step into their own workflow, because the platform will not do it reliably on their behalf.

Start with the next review you begin. Open a protocol file, add a section titled machine-assistance log, and record the tool, model version, prompt, and override rule before you screen the first record. That single file may be the only thing standing between a submitted review and a reproducibility request you cannot answer.