How we rank models

BasedAGI maps real-world workflows (use cases) to the LLMs best suited to them. Each use case defines which benchmarks matter and how much weight each metric carries. Models are scored purely from benchmark measurements — no manual curation, no hidden adjustments.

The result: a use-case-specific score for every model, backed by traceable evidence you can inspect down to the individual benchmark row.

The Utility Score shown on the Model Rankings page is a cross-use-case aggregate:

This confidence-weighted average rewards models that perform well on use cases where the evidence is strong, and discounts scores backed by thin evidence.

Confidence-weighted scoring is more robust than vote-based leaderboards because it preserves uncertainty instead of flattening every judgment into a binary win or loss. A vote-only system rewards marginal, noisy advantages exactly the same way it rewards decisive ones, which makes rankings more sensitive to matchup variance, sample composition, and evaluator noise. By weighting contributions by confidence, the scoring layer can treat weak evidence as weak evidence, so a model climbs because it wins clearly and consistently, not because it accumulates a thin edge across many close calls.

This matters because leaderboards are vulnerable to distortion long before the underlying model quality actually changes. The Leaderboard Illusion argues that benchmark ecosystems can be skewed by selective disclosure and private testing practices, producing rankings that overstate real differences between systems. Confidence-weighted scoring is a practical countermeasure because it reduces the leverage of brittle wins: if the evidence behind an apparent lead is sparse, inconsistent, or low-confidence, the ranking impact should be correspondingly smaller. That makes the scoring surface harder to game and less likely to mistake benchmark noise for genuine product advantage.

For a model-routing product, confidence-weighted scoring also produces better explanations. Users do not just need to know which model ranked first; they need to know whether the margin is trustworthy for their use case. A confidence-aware rank can separate high-signal leaders from provisional leaders, which is materially better than a vote-based table that implies false precision. Inference from The Leaderboard Illusion: if benchmark governance problems can inflate public leaderboard positions, then ranking systems should explicitly encode evidence strength, coverage, and uncertainty rather than treating every comparison as equally reliable.

Citation: The Leaderboard Illusion, arXiv:2504.20879 (Apr 2025)

Each benchmark source receives a reliability_weight from 0.0 to 1.0. In practice, peer-reviewed papers usually land around 0.75-0.90, official leaderboards around 0.80-0.92, and community leaderboards around 0.65-0.80.

Those weights are enforced through the governance pipeline exposed by /ops/source-governance. Sources below 0.7 still contribute, but they contribute materially less to final rankings than higher-trust sources.

Each use case receives an evidence quality grade based on three factors:

Use cases are graded Sufficient, Insufficient, or Unscored. Insufficient rankings still appear but display a warning banner.