What happened
- Five authors, among them David M. Schultz and Alexandra K. Anderson-Frey, published on arXiv on 3 November 2026 a test of six AI tools for scientific literature reviews.
- They evaluated them across four stages: finding papers, interpreting them, synthesizing them and proposing future research.
- The yardstick was a published review of the Spanish plume, a weather pattern, and its corpus of 102 papers.
- In the initial searches, five tools together identified 57 papers and the top one reached 42. With institutional library access, Codex went from 42 to 82 and Claude Code from 15 to 79.
Why it matters
- Many studies test isolated tasks. This one examines the whole process, which is how a review is used in practice.
- The jump with institutional access shows how much the result depends on what the model can read, not only on what it can do.
- A synthesis with wrong quotations or exaggerated consensus can shape a field’s agenda if nobody checks it.
The number
40% to 79%. That is the share of the 84 passing mentions the tools wrongly classified as substantial treatment of the topic. By contrast, the three that received the complete corpus correctly identified the 18 papers that did discuss the Spanish plume in depth.
Context
Two domain experts blindly evaluated the syntheses from three tools. They found them largely accurate but limited in insight, and sometimes containing unsupported claims, quotation errors and exaggerations about the degree of consensus.
The three tools proposed research directions broader than the topic’s literature; two came close to questions identified in the benchmark review. None performed better across all tasks, and the authors give eight recommendations for those who use them. As an arXiv paper, it has not yet gone through peer review.
At Mamífero we have followed this debate. We reported that eight researchers propose studying language models with psychology tests and that rewarding each search step makes an agent more reliable. Also that Pinal designs proteins from a sentence and that six models wrote 148,000 political texts in an experiment on neutrality.
What’s next
- The study’s recommendations point to using these tools to discover and interpret, without giving up the evaluation of evidence.
- Repeating the test on other topics would show whether the result holds outside meteorology.
- Tool versions change quickly, which makes it necessary to repeat measurements like this one.
Bottom line
The tools find more when the library is opened to them, but telling a passing mention from a real contribution remains hard for them. That judgment, according to this work, cannot yet be delegated.
Sources
- Al Kalbani, A., Schultz, D. M., Anderson-Frey, A. K. et al., “Can Large Language Models Review the Scientific Literature? A Test Against a 102-Article Benchmark”, arXiv, 3 November 2026, 2611.05459. Preprint without peer review; this note is based on the abstract.
- Cover image: Vmenkov, CC BY-SA 3.0, via Wikimedia Commons
Edited by Rodrigo Cornejo. How we select and verify: who writes these notes.
