GPQA Diamond

GPQA Diamond is the hardest 198-question slice of the GPQA (Graduate-Level Google-Proof Q&A) benchmark — a set of multiple-choice science questions written by doctoral experts in biology, physics and chemistry, intended to remain difficult even when the answerer is allowed to search the web.^[1][2]

History

GPQA was introduced in a November 2023 paper, GPQA: A Graduate-Level Google-Proof Q&A Benchmark, and published at ICLR 2024. A team of researchers at New York University, Scale AI and Meta posed 448 questions that required genuine subject expertise rather than search skill, and validated them against three groups: domain experts, holders of a bachelor's degree in a nearby field, and skilled non-experts with unrestricted web access.^[1][2]

The headline result defined the benchmark's reputation: PhD-level experts in the relevant field answered correctly about 65 per cent of the time (34 per cent when questions outside their own specialty were included), while skilled non-experts with a web browser reached only 34 per cent — barely above what a strong language model would later score. When the full 448-question set proved noisy, the Diamond subset of 198 questions — those all experts agreed on — became the standard reporting number, and most leaderboards quote GPQA Diamond rather than the full set.^[2][3]

Design

Each question is a multiple-choice problem drawn from graduate and research-level material in biology, physics and chemistry, written and adversarially checked by subject specialists so that answering correctly requires reasoning through a multi-step scientific argument rather than recalling a fact. Because search engines and reference material rarely surface the specific reasoning needed, the benchmark was designed to resist retrieval — hence "Google-proof".^[1]

Scoring is plain accuracy over the 198 Diamond questions. Early results placed GPT-4-based systems at about 39 per cent (roughly the non-expert web-searcher baseline), while frontier models released through the mid-2020s climbed steadily into the 80s and low 90s. By 2026, GPQA Diamond was widely reported as approaching saturation at the top — the strongest systems score in the low 90s, so small differences at the ceiling no longer imply meaningful capability gaps — which pushed evaluation attention toward harder suites such as Humanity's Last Exam, ARC-AGI and agentic benchmarks like Terminal-Bench and FrontierCode.^[3][4]

Recurring criticisms mirror those aimed at MMLU: contamination, since public question sets leak into training corpora; saturation, since expert performance (~65%) and top model performance now overlap; and variance, because a 198-question set gives each item roughly half a percentage point of weight, making one- or two-point differences statistically fragile.^[4]

Applications and Impact

Model cards and release notes from the major laboratories routinely cite GPQA Diamond alongside MMLU as evidence of scientific reasoning ability, and aggregate indexes such as Artificial Analysis's Intelligence Index include it as one component of a composite score. Researchers also use it to study reasoning-versus-recall behaviour: models that improve most on GPQA Diamond when allowed a thinking budget are typically improving at multi-step deduction rather than at fact retrieval.^[2][4]

>See Also

• AI Benchmarking
• MMLU
• Humanity's Last Exam
• ARC-AGI
• Reasoning Models
• LLM Evaluation