KINA: Mapping the Knowledge Boundary of Frontier AI
As frontier AI systems become more capable, knowledge evaluation is becoming harder to read. Better retrieval, tool use, and agent workflows can make systems appear more competent, but they do not erase the importance of the underlying model's disciplinary knowledge. That internal knowledge still shapes how well an AI system interprets evidence, connects concepts, and reasons across unfamiliar domains.
At the same time, stronger models have made benchmark construction more fragile. Many early test sets are close to saturation, so it is tempting to respond by making questions larger, harder, or more specialized. But difficulty alone is not enough. A question can be obscure without being representative, and specialized questions become harder to verify. The next generation of knowledge benchmarks needs to be difficult, representative, source-grounded, and auditable.
KINA, the Knowledge Index of Noah's Ark, is built for this purpose. Rather than simply making questions harder, KINA focuses on disciplinary representativeness and cognitive depth. It asks whether models can handle source-grounded questions that reflect the central concepts, methods, and reasoning patterns of fine-grained academic disciplines.
KINA contains 899 ten-option pseudo-multiple-choice questions across 261 fine-grained subfields, 70 fields, and 12 top-level disciplines. Each item passes through a four-stage verification pipeline: rule-based screening, double-blind expert review, LLM-as-judge consensus, and agentic refinement. The goal is not only to rank models, but to map where frontier AI systems possess reliable disciplinary knowledge and where that knowledge remains thin, uneven, or brittle.
The result is a more sober view of frontier intelligence: even today's strongest systems remain far from saturation when evaluated against a rigorous, multidisciplinary taxonomy.
1. The Problem: Scaling is Hiding Our Ignorance
For years, the field has operated under the assumption that "more is better"—more scale and more harder questions. However, simply escalating difficulty (as exemplified by benchmarks like HLE) often produces "esoteric trivia." This creates a benchmark that is difficult but not diagnostic; it tests whether a model has memorized obscure facts, not whether it has internalized the causal structures and core theoretical pivots of a discipline. KINA is motivated by three gaps in current knowledge evaluation:
1.1 Disciplinary Neglect and Lack of Representativeness
Most existing benchmarks suffer from severe over-fitting to high-resource domains (e.g., Biology, Physics, Chemistry in GPQA). By focusing on a narrow subset of popular fields, they neglect the "vast specialized territories" of human knowledge—such as Veterinary Medicine, Metallurgy, or Oceanography. This Disciplinary Neglect means we lack a comprehensive diagnostic anchor for the LLM’s true cognitive manifold. A benchmark that does not map the full topology of human knowledge cannot measure whether a model has mastered the core competencies and central theories of a specific field.
1.2 Data Contamination and "Lossy" Reasoning
A persistent vulnerability in static evaluation sets is Data Contamination. Because benchmarks like MMLU are largely scraped from publicly available web sources, models are often bypassing genuine deductive reasoning through "superficial memorization." They are effectively performing high-speed retrieval of answers they have already seen during pre-training. This results in severe "hallucinatory failure modes," where models appear to reason but are merely pattern-matching against contaminated training corpora.
1.3 The "Lazy Consensus" Trap
Perhaps the most damaging failure is the reliance on blind-trust annotation workflows. When evaluating frontier intelligence, conventional crowdsourcing or even expert-in-the-loop pipelines often default to human "lazy consensus." Annotators, incentivized by economic payoff rather than quality, frequently "rubber-stamp" flawed, ambiguous, or idiosyncratic trivia. This failure of quality control means that the benchmark itself is scientifically unreliable. As demonstrated by independent post-release audits of high-difficulty benchmarks like HLE, a significant percentage of questions are factually incorrect, making headline scores harder to interpret without careful quality control and post-release auditing.
2. KINA's Pipeline
KINA represents an engineering departure from the "blind-trust" annotation workflows that have long plagued benchmark development. Rather than assuming annotators are infallible experts, we treat the annotation process as a constrained engineering problem. Our objective is to structure the workflow such that rigorous, independent evaluation is the most efficient path for the contributor, effectively mitigating the common tendency toward "lazy consensus."
2.1 The Four-Stage Verification Pipeline
To ensure high knowledge density and representativeness, every instance in KINA must survive a rigorous 4-stage verification pipeline.
Stage 1: Rule-Based Review (Automated Scripts): We filter the initial submission for structural integrity, ensuring that LaTeX is compiled flawlessly in Markdown, options are mutually exclusive, and the cosine similarity of text embeddings remains below 0.8 to prevent duplicate content. This stage also applies our Hardness Filter, where only questions that at least 3 out of 5 flagship LLMs fail to answer correctly are retained.
Stage 2: Expert-Based Review (Double-Blind): Two independent domain experts evaluate each submission. By assigning items to two independent reviewers under double-blind allocation, we reduce information leakage between reviewers and make superficial agreement easier to detect through later audit.
Stage 3: LLM-based Filtering (LLM-as-a-Judge): we deploy a consensus voting mechanism where three independent LLMs evaluate the instance for multi-dimensional features, including uniqueness and potential socio-economic impact.
Stage 4: Agentic refinement (flagged items): Finally, to address boundary defects like context drift and ambiguous causal chains in flagged test items, a model-agnostic two-agent refinement loop (instantiated with GPT-5.2-Pro in this run) is implemented: a web-enabled upstream agent searches for counter-evidence, a downstream agent revises the stem or adds missing premises, and revised items must pass human review under the Stage 2 rubric before re-entering the dataset pool.
2.2 Incentive-Aligned Review
KINA’s expert review stage is built around a bonus-on-bar tournament. Two reviewers score each item independently. The reviewer with the higher validated score receives a bonus only if the winning score clears a quality threshold. This design changes the review incentive: reviewers are rewarded not for quick approval, but for careful, high-quality evaluation.
The mechanism is paired with stochastic audits on approved items. If an item later fails audit, the review history makes it possible to identify where the review process broke down. This does not make the pipeline immune to all forms of error, but it makes review quality more measurable, accountable, and improvable than a flat-payment workflow.
In the paper, this mechanism is analyzed formally as a way to improve released review quality relative to flat per-item payment under stated assumptions. In practice, it is combined with discipline-specific reviewer qualification, double-blind allocation, audit sampling, and human confirmation after refinement.
2.3 The Data Manifold: Suppressing Guessing
To ensure the benchmark remains diagnostic, we impose strict constraints on the data manifold itself:
10-Option Combinatorial Format: By increasing the option count from the standard 4 to 10, we mathematically suppress random guessing (reducing chance accuracy from 25% to 10%).
Pseudo-Multi-Choice Structure: Each question requires joint reasoning over multiple complex statements. This forces the model to move beyond simple pattern matching or process-of-elimination shortcuts.
3. Experimental Benchmarking: A Reality Check for SOTA Models
We subjected 42 models from 13 major AI labs to the KINA gauntlet, spanning a diverse spectrum of architectures—from proprietary flagship models to open-source reasoning specialists. The results are humbling, forcing a recalibration of our current expectations for "frontier intelligence."
3.1 The "Omniscience" Fallacy: Nowhere Near Saturation
KINA reveals we are far from saturation. Even the most capable frontier model, Gemini-3.1-Pro-Preview, achieves an overall accuracy of merely 53.17%. When we push models to move beyond simple pattern recognition into deep disciplinary mastery, the performance "drops" significantly. This confirms that multidisciplinary mastery remains an open research challenge, and traditional benchmarks have been masking significant knowledge gaps in frontier systems.
3.2 Tool Use: Consistent but Uneven Gains
Tool use improves performance, but not uniformly. In the five-model web-search cohort, all evaluated models gain from search, with improvements ranging from +1.50 to +5.17 points. The pattern is non-monotonic: weaker models may use retrieval to fill parametric knowledge gaps, while stronger models can better synthesize retrieved evidence into a coherent reasoning chain. Because the cohort is small, this pattern should be read as suggestive rather than universal.
3.3 Disciplinary Variance: Hard Science Plateaus vs. Soft Science Spreads
KINA’s 261-discipline taxonomy allows us to decompose model behavior structurally. We observe a stark contrast between "hard" and "soft" sciences:
Hard Science Plateaus: In Science and Engineering, model performance plateaus uniformly across the frontier, suggesting these fields are well-represented in pre-training data and amenable to systematic reasoning chains.
Soft Science Variance: In disciplines like Sociology, Management, and History, we observe tier-dependent spreads of up to 38 points. This indicates that contextual and interpretive knowledge is internalized less reliably than formal reasoning.
3.4 Scaling Laws and The Knowledge Density Metric
We also examine scaling behavior in the Qwen3 and Qwen3.5 families. Qwen3 models scale at roughly 7.6 points per decade of parameters, while Qwen3.5 reaches about 14.8 points per decade. This is consistent with the view that data curation, training recipe, and knowledge density matter alongside parameter count. At matched active parameter counts, MoE models outperform dense models, suggesting that sparse activation may be beneficial for knowledge-intensive tasks. These trends are suggestive rather than definitive, since model-family and post-training differences remain possible confounders.
4. Deep Dive: What the Data Tells Us
4.1 The High-Low Boundary: Decoding Domain Complexity
We analyzed the High and Low boundary indices of the top-10 models. This reveals a clear structural split:
Hard-Science Plateau (Science, Engineering, Agronomy): These disciplines exhibit remarkably low variance (e.g., Science Δ= 9.83). Models converge toward a performance plateau here. This suggests that these domains are well-represented in existing pre-training corpora and are inherently amenable to systematic reasoning chains and formal axiom-based deduction.
The Soft-Science Variance (Sociology, Management, History): Conversely, fields like Sociology show a massive discrimination index (Δ= 38.16). This high coefficient of variation (CV > 30%) indicates that model performance is strongly tier-dependent. It suggests that the contextual, interpretive knowledge required for "soft sciences" is significantly harder for models to acquire systematically during pre-training compared to formal logic.
4.2 The Atypical Profile Phenomenon Individual model performance profiles often defy simple scaling laws. For instance, Claude-Opus-4.6 shows an anomalous disciplinary signature: it ranks remarkably high in Philosophy (36.54%) and Agronomy (61.88%), yet underperforms in History (15.38%).
This uneven distribution suggests that knowledge coverage is non-uniform even among frontier systems. Instead of a monolithic intelligence that scales equally across all fields, we are seeing models with "specialized knowledge islands." This highlights the utility of KINA’s fine-grained breakdown: it exposes the specific knowledge holes in proprietary systems that aggregate metrics (like ALL) would otherwise obscure.
4.3 Subject-Level Stability vs. Volatility
By visualizing the density of subject-level scores, we gain insights into the "stability" of a model’s knowledge.
Models with right-shifted, narrow ridgelines (e.g., top-tier Gemini/Claude models) exhibit high and stable performance across the knowledge manifold.
Models with broad, flat distributions indicate that the model's knowledge is "brittle"—it might perform excellently on some niche subjects but fail catastrophically on others, even within the same disciplinary cluster.
4.4 Intersections of High-Performance
We also examine how high-performing items overlap across disciplines. The result suggests that strong performance in one disciplinary cluster does not necessarily imply strong performance in another. KINA therefore provides a way to study not only overall model strength, but also the structure of model expertise.
5. Verdict: From Ranking Models to Mapping Knowledge
The era of "brute-force" benchmarking is ending. KINA demonstrates that the primary bottleneck in evaluating frontier intelligence is not the lack of "hard" questions, but the lack of diagnostic rigor. We must pivot from benchmarks designed to rank models to those designed to map the knowledge manifold.
5.1 Beyond Aggregate Accuracy
KINA suggests that the next stage of knowledge evaluation should move beyond a single aggregate ranking. Ranking remains useful, but it is not enough. A diagnostic benchmark should also show where models converge, where they diverge, and which disciplines expose persistent gaps in model knowledge.
For Practitioners: Our results show that model performance is highly structural. If your model struggles in History or Law but excels in Engineering, it’s rarely a parameter count issue—it’s a Disciplinary Representativeness issue. The model lacks the structural knowledge to connect core theories, not just the capacity to store data points.
For Researchers: The game-theoretic annotation framework is KINA’s most significant contribution to the community. We urge future benchmark designers to adopt this zero-trust approach. By aligning reviewer incentives (the "Payoff Matrix"), we ensure that high-quality, high-effort review is systematically rewarded, effectively insulating the dataset from the "lazy consensus" and human annotation biases that currently plague the field.
5.2 Agentic Refinement for Benchmark Quality
KINA also uses agentic workflows during dataset construction. The agentic refinement loop is not intended to replace expert review. Instead, it helps identify counter-evidence, missing premises, context drift, and logical vulnerabilities before an item is released. Human reviewers then confirm whether the revised item satisfies the same quality criteria as other accepted items. This points to a broader direction for benchmark construction: future benchmarks may become more iterative, auditable, and self-correcting, while still keeping human domain expertise at the center of final validation.
5.3 Final Outlook: Auditing Machine Intelligence
Noah’s Ark was built to preserve life when the storm came. KINA is built to preserve the integrity of human knowledge when the AI "data flood" arrives. We are moving past the point where a single aggregate accuracy score (like avg@4) tells us anything meaningful. As benchmarks become saturated, the value lies in the structural findings: identifying where models plateau and where they spread. We are not just grading models anymore; we are auditing the architecture of machine intelligence itself.
