Read an AI paper as a chain of evidence: a precise claim, a method, an evaluation and a set of limitations. You do not need to understand every equation on the first pass to discover whether the headline matches the experiment.
First pass: find the actual claim
Start with the abstract, introduction, main results and limitations. Write the central claim in one sentence using the paper’s conditions: “Method A improves metric B on dataset C under budget D.” If you can only write “the model is smarter”, the claim is still too broad.
Identify the publication status and version. A preprint can contain valuable work, but appearing on a preprint server is not the same as journal or conference peer review. A revised version may also change a result quoted in an earlier news report. Keep the identifier and version with your notes.
Check what the test data can establish
Training, validation and test data serve different purposes. Look for whether the authors separated them, how they selected examples and whether the model could have encountered test material during training or development. Public benchmark exposure is a concern to investigate, not automatic proof of contamination.
Ask whether the evaluation resembles the intended use. Multiple-choice accuracy does not by itself establish reliable open-ended reasoning. A robot tested in one controlled environment has not demonstrated general operation in unfamiliar homes. A scientific predictor evaluated on closely related samples may behave differently under distribution shift.
Compare the budget as well as the score
A baseline is only useful when the comparison is understandable. Check model versions, prompt design, available tools, retrieval access, context length and the number of attempts. An improvement can come from a method, more computation, better data or a combination.
| Claim in a headline | Question for the paper |
|---|---|
| “Beats the leading model” | On which tasks, versions and inference budgets? |
| “Solves difficult problems” | One attempt or many? How is success verified? |
| “Uses less compute” | Training, inference or both? Does the accounting include data preparation and repeated runs? |
| “Generalizes to new settings” | What changed between development and testing? Were the new settings selected independently? |
| “More reliable” | Across how many runs, and what failures remain? |
Read the uncertainty and failure cases
A single number hides variation across tasks, seeds and samples. Look for confidence intervals, repeated runs or other uncertainty reporting appropriate to the study. Small changes may not support broad conclusions, especially when many variants were tried and only the best is reported.
Inspect ablations: removing or changing a component helps test which parts matter. Then read the failure examples and limitations. A limitation is not necessarily a reason to dismiss the work; it tells you where the evidence stops.
If evaluation uses a model as a judge, check the rubric, judge version, answer order and whether people or independent checks validate its judgments. Agreement with a judge is different from direct evidence that a real-world task succeeded.
Can someone else check the result?
Look for code, data access, model weights, dependencies, seeds and exact evaluation commands. “Code available” can mean a usable reproduction package or a repository missing important pieces. Check the license and practical compute requirements as well as the link.
Distinguish reproducing the authors’ experiment from independently verifying the wider claim. Running released code on the same test set is useful, but external data or a new deployment can expose limitations that the original evaluation did not address.
A reusable reading note
- Identity: title, authors, source URL, version and publication status.
- Claim: one sentence including the task and conditions.
- Evidence: datasets, baselines, metrics, budgets and uncertainty.
- Boundaries: reported failures and assumptions that may not hold elsewhere.
- Reproduction: available artifacts and what remains inaccessible.
- Next check: one concrete experiment or source needed before relying on the claim.
This structure also improves a news summary: state what was measured, link the paper and say what has not yet been shown. Follow AI research news and AI for science for papers to practice on.
Sources and scope
- arXiv submission guidelines — submission and publication context; arXiv distribution should not be equated with peer review.
- Hugging Face model-card documentation — a practical structure for training information, intended use, limitations and evaluation.
- NIST AI Risk Management Framework — evaluating AI in its context of use.
The checklist is an editorial synthesis for readers, not a substitute for a field-specific technical review. It does not award a quality score to papers. Send corrections or additional references.
Follow the latest coverage
1000 Years of AI Safety
A three-year AI pause could buy 1000 years' worth of AI safety work. See: https://1000yearsofaisafety.com How much AI safety work a pause produces, in years of the whole 2025 AI safety field's work . The government bans new large training runs and gets the top three US labs to redirect their GPUs and researchers to safety research. We developed a model of an AI pause that estimates how much AI safety research it produces.
A Tourist Mathematician’s Apologia: Living in a Human-AGI World
Graham Greene described G. H. Hardy’s 1940 book, A Mathematician’s Apology , as the best account of what it is like to be a creative artist. Hardy had many friends, wide-ranging interests that included the game of cricket, and empathy for the less well-off. But it was his creative work in pure mathematics that gave Hardy “the one great permanent happiness of my life.” “Mathematics,” writes C. P.
Human vs AI in Go in 2026: Lessons for superintelligence?
In July 2026, Korean Go player Shin Jin-seo defeated KataGo 2-1 , with only [1] a 2-stone handicap (favoring Shin). I thought this was very surprising, and posted on LessWrong and Twitter asking how to interpret this. I didn’t get any entirely satisfactory answer, so I decided to spend ~one day looking into it and writing up my understanding.