Can LLMs Run Your Systematic Review? What 51 Recommendations Require

If you are deciding whether ChatGPT-class tools can run your systematic literature review, the most direct answer available right now comes from the team that built one of the most explicit checklists for the question. In the GUEST preprint (GenAI Use and Evaluation in SLR Tasks), the authors state their conclusion plainly: “Current empirical evidence indicates that GenAI cannot be relied upon to perform unsupervised systematic reviews so requires human oversight. Nevertheless, it can offer cost-effective assistance for repetitive screening tasks and secondary validation.”

That single sentence does more work than most of the debate around AI-assisted evidence synthesis. It forbids one thing (autonomous reviews) and permits another (supervised help with the tedious parts). Everything else in this article, including the stage-by-stage table below, works out that boundary in practice, plus the reporting burden that comes with it. Every number and category in the framework is author-reported from a preliminary preprint, and I will flag that where it matters.

The practical consequence for authors and the reviewers judging them: cheap screening is now defensible, but the cost of claiming rigour has moved onto you. You must document the tool, the prompts, the validation evidence, and the retraction status of every citation the tool surfaced.

What GUEST actually is

GUEST is, in its authors’ words, “a comprehensive framework of 51 recommendations designed to guide researchers through planning, executing and transparently reporting GenAI-assisted literature reviews and validation studies.” The count of 51 and the structure are the authors’ own, not an external audit.

The framework organizes its guidance into four categories, which the authors describe as “spanning SLR practice adjustments, general secondary-study standards, GenAI evaluation practices and core empirical rigour.” Those four categories are the natural rows of any decision table built from this work, and I use them that way below.

Two things about the framing are worth noticing before you adopt it. First, GUEST explicitly anchors its reporting expectations to existing standards: “SEGRESS (Kitchenham et al., 2023) for software engineering, and PRISMA 2020 (Page et al., 2021) for evidence synthesis in a broader context.” That gives editors a named, already-legitimate standard to demand rather than inventing AI-disclosure rules from scratch. Second, the authors are software-engineering researchers writing from software-engineering review practice. Whether the framework transfers cleanly to health evidence synthesis, where the stakes and the regulatory environment differ, is unverified in this preprint and should be checked before you treat it as authoritative outside its home discipline.

How the guidelines were built, and why that limits them

GUEST is not the product of controlled trials. According to the preprint, it was assembled from a rapid review of existing guidelines for using and evaluating GenAI in SLRs, plus thought experiments, literature guidance, and the authors’ own experience conducting reviews and evaluating tools. The authors themselves concede that guidelines of this kind “can rest on expert judgement rather than reported evidence,” and they tried to compensate by synthesizing failure modes from two empirical companion studies that applied GenAI tools to SLR tasks.

That is a reasonable mitigation, and it is also the reason to hold the framework at arm’s length where it makes strong claims. The “cannot be relied upon” conclusion rests on the authors’ synthesis of the current literature, not on a trial reproduced here. That does not make it wrong. It makes it a judgment you should treat as provisional and weigh against independent accuracy evidence as it accumulates; an independent published benchmark of this task, discussed below, points the same way.

The stage-by-stage delegation table

The table below is the working decision tool. Two honesty notes before you use it. GUEST does discuss the stages at a task level: it names “literature screening, data extraction, qualitative synthesis, and risk-of-bias and strength-of-evidence assessment” as the SLR-specific tasks its first category of recommendations covers. What it states as a verdict is the whole-review position, with a carve-out for screening and secondary validation. The per-stage “unsupervised” entries below extend that verdict, which is why every row in that column except screening carries an “inference” tag. The disclosure column reflects what GUEST’s alignment with PRISMA 2020 and SEGRESS implies, read alongside what a thorough human-conducted review already reports.

The pattern across rows is consistent: GenAI may touch every stage, but it owns none of them. The GUEST position does not ask you to keep AI out of the review; it asks you to keep a named, accountable human inside every stage of it.

What good disclosure already looks like

The reporting bar GUEST extends is not hypothetical. A PRISMA-guided systematic review of machine learning for diffuse reflectance spectroscopy names its eight databases (PubMed, IEEEexplore, Nature, ACM Library, Scopus, Web of Science, CINAHL, Embase), states that “the last search was conducted on the 7th of May 2024,” restricts results to post-2004 publications, and notes that “databases were identified in collaboration with a library information specialist.” That is the human baseline: a named search strategy, a date stamp, a scope decision, and a specialist’s involvement.

A more recent systems-security SLR on eBPF-based cybersecurity mechanisms shows the completeness end of the same bar: “The complete list of 54 primary studies analyzed is provided in Tables 3 and 4,” with the PRISMA screening process documented in a dedicated figure.

GUEST’s disclosure demands sit on top of this. A reviewer applying the framework can reasonably ask for language along these lines in a methods section:

Screening was assisted by [tool, version] using the prompts reproduced in Appendix A. The model’s include/exclude suggestions on a random sample of [N] records were checked against dual human screening; disagreement was [X%] and all final decisions were made by the human reviewers. All included citations were checked for retraction status on [date] via [source].

None of those bracketed items requires a new standard. PRISMA 2020 and SEGRESS already expect the search and screening detail; GUEST adds the tool, prompt and validation layers. If an author cannot fill in the brackets, the review is not GUEST-compliant, and an editor now has a citable framework to point to when saying so.

The retraction trap applies to humans too

One failure mode deserves its own section because it cuts against the instinct to treat GenAI as the sole citation-integrity risk. A study of retracted citations found that retracted papers “are often cited in systematic reviews and meta-analyses” across computer science and biomedical research, and identifies a major cause: “the authors are not informed about the status of article retractions, either because they do not receive enough notifications in journals and databases or because they depend on saved copies or find uncorrected versions available on open-access platforms.”

The same study quantifies the scale of the corpus behind that finding (source): 26,528 filtered documents, of which 4,924 were categorized as retracted for plagiarism and 6,420 for fake peer review. The 156 papers carrying both retraction reasons were excluded (source); those 156 had accumulated 1,954 citations. It also reports that plagiarism-related retracted papers receive 2.5 times more citations than fake-peer-review retractions despite fewer papers in that category, meaning flawed sources compound once they enter circulation.

Two conclusions follow. First, the all-human review is not a clean control. Retraction leakage already happens without any AI in the loop, so editors should demand citation-status checks from every submission, not just the GenAI-assisted ones. Second, GenAI changes the risk profile rather than creating it: any tool that surfaces papers from saved copies, preprint servers or uncorrected open-access versions inherits exactly the notification failure the study describes. A retraction-status check on every AI-surfaced citation is a cheap, concrete step with direct evidence behind it.

Who verifies the output is a design decision

“Human oversight” sounds like a checkbox. It is not. A qualitative study of 14 graduate students evaluating AI-generated content found that “students construct evaluation frameworks shaped by three main factors: professional identity, verification capabilities, and system navigation experience. Rather than uniformly accepting or rejecting LLM outputs, students protect domains central to their professional identities while delegating others.”

Fourteen participants is a small, qualitative sample, so treat this as a mechanism, not a prevalence estimate. But the mechanism has an operational consequence for review teams: the person you assign to verify GenAI screening decisions will verify some things carefully and wave others through, depending on what they consider their home turf and how practiced they are at checking machine output. A junior researcher delegated to “run the AI screening” may be precisely the person least equipped to catch its errors in the domain that matters most.

The fix is structural, not motivational. Assign verification of each stage to someone whose expertise overlaps that stage, and record who verified what. That turns GUEST’s “human oversight” requirement from an assurance into an audit trail.

What a validation study could look like

GUEST’s “GenAI evaluation practices” category implies that teams should be able to point to evidence that their tool worked on their task, not just that a vendor says it works. The framework itself does not amount to a ready-made validation protocol, but an independent benchmark of exactly this task now exists, and its numbers are worth having in front of you before you delegate anything. The SESR-Eval dataset contains “34,528 labeled primary studies, sourced from 24 secondary studies published in software engineering (SE) journals,” and its authors “benchmark title-abstract screening using 9 LLMs.” Their summary is blunt: “no LLM managed a high recall with reasonable precision.” The authors report screening accuracy across the secondary studies ranging from 0.34 to 0.85, and the strongest results sit around 0.56 to 0.60 precision while recalling only 0.42 to 0.46 of the eligible studies. Cost is not the barrier: the authors put LLM screening at less than $40 per secondary study even for the most expensive model (source). That is independent evidence pointing the same way as GUEST’s conclusion.

For teams designing their own validation, the closest published template is adjacent, and I flag it as an analogy, not SLR evidence: the ATANT evaluation framework, built for a different AI property entirely, combines “a formal definition of continuity (7 properties), a model-independent evaluation methodology (10 checkpoints), a narrative test corpus (250 stories, 1,835 questions), and a sequenced compliance framework (4 levels)” (source).

The transferable idea is the shape, not the content: define the property you care about independent of any one model, build a fixed test corpus with known answers (a set of records whose include/exclude status is already adjudicated), and grade compliance in levels rather than pass/fail. A screening validation on that pattern would run the tool against a few hundred pre-adjudicated abstracts before trusting it on the full corpus, and would report the result. A public benchmark does not remove the need for that local check: SESR-Eval’s authors found that “the differences in screening accuracy between secondary studies are greater than differences between LLMs,” so a model’s benchmark average does not tell you how it will behave on your corpus. The local check is the validation evidence that speaks to your review.

The delegation rule I would apply, consistent with the evidence here: no SLR stage runs unsupervised. GenAI earns its place as a supervised, cost-saving assistant on repetitive screening and secondary validation, the two uses the GUEST authors themselves endorse. Everything else gets a named human with relevant expertise, documented verification, and disclosure written against PRISMA 2020 or SEGRESS with the tool, prompts and validation sample made visible.

Be skeptical of two framings you will encounter. “AI-accelerated systematic review” products need reading against the line GUEST draws: the framework permits supervised assistance and forbids autonomous review, and where marketing conflates the two it is selling you the prohibited version under the permitted one’s name. And “AI reviews fabricate citations” narratives overstate the novelty: the retraction evidence shows human workflows already leak compromised sources, which is an argument for universal citation-status checks, not for singling out tool users.

The honest limitation sits with the framework itself. The 51 recommendations, the four categories and the “cannot be relied upon” conclusion are all author-reported, drawn from a rapid review, thought experiments and the authors’ own software-engineering review experience rather than controlled trials. An independent published benchmark of this task, SESR-Eval, is software-engineering-only, and its recall figures back the caution rather than challenge it. Transfer to health evidence synthesis is unverified. Use GUEST as the best-organized checklist currently available, cite it as preliminary, and recheck for a peer-reviewed revision before you build a lab policy or an editorial requirement on it.

Frequently Asked Questions

What is the GUEST framework and how many recommendations does it contain?

GUEST is, in its authors’ words, “a comprehensive framework of 51 recommendations designed to guide researchers through planning, executing and transparently reporting GenAI-assisted literature reviews and validation studies.”

Can GenAI be used for title and abstract screening in systematic reviews?

Explicitly endorsed carve-out: “cost-effective assistance for repetitive screening tasks and secondary validation,” with human oversight

What should be disclosed when using GenAI for screening in a systematic review?

Tool, version, prompts, inclusion criteria given to the model, sample-checked agreement, final human decisions