- Type
- Evaluation methodology
- Key benchmarks
- MMLU, HumanEval, GSM8K, HELM, BIG-Bench, AIME
- Maintained by
- Academic labs, Hugging Face, EleutherAI, NIST
- Key use
- Model comparison, capability tracking, safety evaluation
- Related
- Large language models, reasoning models, AI safety, model serving
- Type
- Evaluation methodology
- Key benchmarks
- MMLU, HumanEval, GSM8K, HELM, BIG-Bench, AIME
- Maintained by
- Academic labs, Hugging Face, EleutherAI, NIST
- Key use
- Model comparison, capability tracking, safety evaluation
- Related
- Large language models, reasoning models, AI safety, model serving
AI benchmarking is the practice of evaluating artificial intelligence systems using standardised datasets, tasks, and scoring metrics to measure their capabilities, compare them against other systems, and track the progress of the field over time. Benchmarks provide the empirical foundation for claims about AI capability, enable researchers and practitioners to select appropriate models for specific applications, and help safety researchers identify failure modes and limitations.
As AI systems have grown in scale and generality, benchmarking has evolved from narrow task-specific tests (accuracy on a particular dataset) to broad multi-task suites that probe reasoning, factual knowledge, coding ability, safety, and alignment. In 2025, frontier model benchmarking encompasses hundreds of distinct evaluations, and the benchmark ecosystem itself has become a subject of methodological scrutiny.
Purpose and Importance
Benchmarks serve several distinct functions in the AI ecosystem. For research, benchmarks provide common points of comparison across papers and organisations, enabling the field to measure whether new techniques represent genuine improvements or merely dataset-specific optimisations. For model selection, practitioners use benchmarks to choose models for specific applications. For safety and alignment, benchmarks measure model behaviour on harmful content generation, instruction following, truthfulness, and consistency. For procurement and governance, enterprises and governments increasingly require benchmark disclosures when acquiring AI systems.
General Language Understanding
MMLU (Massive Multitask Language Understanding, Hendrycks et al., 2020) evaluates models across 57 subjects including mathematics, law, medicine, history, and computer science, drawn from academic examinations. MMLU became the most widely cited general knowledge benchmark for LLMs. MMLU-Pro (Wang et al., 2024) is a harder variant with 10-choice questions instead of 4, causing a 16-33 percent accuracy drop compared to the original benchmark and addressing ceiling effects.
BIG-Bench (Srivastava et al., 2022) is a collaborative benchmark comprising over 200 tasks designed to probe capabilities beyond standard language understanding. BIG-Bench Hard selects the 23 most challenging tasks for evaluation.
HELM (Holistic Evaluation of Language Models, Liang et al., 2022) from Stanford assesses models across multiple scenarios and metrics simultaneously, including accuracy, calibration, robustness, fairness, and efficiency, providing a multi-dimensional profile rather than a single score.
Coding and Mathematical Benchmarks
HumanEval (Chen et al., 2021) evaluates code generation through 164 Python programming problems, testing functional correctness using unit tests. It remains the most widely cited coding benchmark. MBPP (Mostly Basic Python Programs) and LiveCodeBench — which sources problems from competitive programming contests to avoid contamination — provide complementary evaluations.
GSM8K (Grade School Math, Cobbe et al., 2021) contains 8,500 grade-school mathematics problems requiring multi-step reasoning. MATH (Hendrycks et al., 2021) is a harder dataset of competition-level mathematics.
AIME (American Invitational Mathematics Examination) problems have become the standard frontier benchmark for mathematical reasoning as of 2025. Qwen3.5-plus scored 91.3 percent on AIME 2026, and GPT-5.3 Codex achieved 94 percent on AIME 2025, reflecting the dramatic capability gains of reasoning models trained with reinforcement learning.
Reasoning, Knowledge, and Safety
ARC (AI2 Reasoning Challenge) distinguishes easy and challenge sets of science questions. HellaSwag evaluates commonsense natural language inference. TruthfulQA measures the tendency of models to generate truthful rather than plausible-sounding false answers. WinoGrande tests commonsense reasoning via pronoun resolution.
Humanity's Last Exam (HLE, 2025) is a crowdsourced benchmark of over 3,000 expert-level questions across diverse academic disciplines, designed to be unsolvable by generalist AI and targeting the frontier of model capability. Safety-focused benchmarks such as HarmBench, SORRY-Bench, and WildGuard probe model behaviour on harmful content generation and instruction following in adversarial settings.
Multimodal Benchmarks
MMBench, MMMU (Massive Multidiscipline Multimodal Understanding), and MMStar evaluate vision-language models across image understanding, chart interpretation, and scientific reasoning. DocVQA and TextVQA test reading comprehension within document images. Video-MME assesses understanding of video content.
Benchmark Contamination and Limitations
A critical methodological concern is benchmark contamination: if a model's training data contains the questions or answers from a benchmark, its reported performance overstates true capability. As benchmarks are published, their problems gradually appear in internet crawls and subsequent training sets, inflating scores over time. Dynamic or continually updated benchmarks — such as LiveCodeBench and Chatbot Arena (which uses real user preferences rather than fixed test sets) — partially address this problem.
Goodhart's Law applies acutely to AI benchmarking: as a benchmark becomes a target for optimisation, it ceases to be a good measure of the underlying capability it was designed to probe. Benchmark scores also fail to capture deployment-relevant properties such as latency, cost, context length handling, and safety behaviour across diverse user populations.
AI benchmarking is becoming relevant to Malaysian enterprises and government agencies as they evaluate and procure AI systems. The National AI Office Malaysia and MDEC have begun developing guidelines for responsible AI procurement, with benchmark transparency — requiring vendors to disclose evaluation results on standard tests — as an emerging expectation for public sector AI deployments.
Malaysian universities, including Universiti Malaya (UM), Universiti Teknologi Malaysia (UTM), and Universiti Putra Malaysia (UPM), publish research evaluating model performance on benchmarks relevant to Bahasa Malaysia and Southeast Asian contexts, including translation quality on Malay-English pairs and factual accuracy about Malaysian geography, law, and culture. Existing global benchmarks such as MMLU do not adequately test knowledge relevant to Malaysian legal, regulatory, and cultural contexts, motivating the development of localised evaluation sets.
MDEC's AI for Industry programme, which targets adoption in manufacturing, logistics, and financial services, is beginning to incorporate benchmark requirements into AI vendor qualification criteria. Vendors seeking to deploy AI solutions in regulated sectors — banking, healthcare, insurance — are expected to provide evidence of performance on relevant capability and safety benchmarks as part of their compliance documentation submitted to BNM, SC, and the Ministry of Health.
The ASEAN AI governance context is also relevant: the ASEAN Guide on AI Governance and Ethics (2024) encourages member states to develop shared evaluation frameworks for AI systems deployed across the region. Malaysia's participation in these multilateral discussions, through the National AI Office and MDEC, positions the country to influence regional benchmark standards that reflect Southeast Asian linguistic, cultural, and regulatory diversity.
For Malaysian AI startups seeking to commercialise models or AI-powered products, credible benchmark results are increasingly important signals for investors, enterprise buyers, and government procurement officers. Organisations such as Cradle Fund and MDEC's Malaysia Tech Entrepreneur Programme (MTEP) are beginning to ask portfolio companies to report standard benchmark performance as part of technology due diligence.
See Also
References