Supply Chain

Vendor contract summary (procurement)

Summarize vendor contracts into key terms, risks, and deviations.

task.summarize_legal_contracttask.contract_term_extraction

Evidence quality is currently limited for this use case. Rankings below are useful for exploration, not a strong winner claim.

Provisional leader

gemini-2.5-pro

Current leader based on limited benchmark evidence. Treat this ranking as directional until coverage improves.

27.0%

Best benchmark score

39.7%

Confidence

All ranked models — top 3

🥇

gemini-2.5-pro

27.0%

🥈

gpt-5-2025-08-07

26.6%

🥉

gemini-3.1-pro-preview

25.7%

Ranked Models

Evidence Quality

82%

Evidence Points

Top Signal

FACTS Benchmark Suite: facts_grounding_score_pct

All Ranked Models

30 of 30 models

Rank	Model	Score	Confidence	Price / 1M	Evidence sources
🥇	gemini-2.5-pro Strong on FACTS Benchmark Suite facts_grounding_score_pct and Vectara HHEM Leaderboard overall_hallucination_error_pct	27.0%	40%	$3.44	FACTS Benchmark SuiteVectara HHEM Leaderboard
🥈	gpt-5-2025-08-07 Strong on SciArena Leaderboard rating_elo and FACTS Benchmark Suite facts_grounding_score_pct	26.6%	35%	—	SciArena LeaderboardFACTS Benchmark Suite
🥉	gemini-3.1-pro-preview Strong on Vals Finance Agent overall_accuracy_pct and FACTS Benchmark Suite facts_search_score_pct	25.7%	29%	$4.50	Vals Finance AgentFACTS Benchmark Suite
#4	gpt-5-mini-2025-08-07 Strong on LEXam Leaderboard average_score_pct and SciArena Leaderboard rating_elo	24.6%	39%	—	LEXam LeaderboardSciArena Leaderboard
#5	gemini-3-pro-preview Strong on SciArena Leaderboard rating_elo and Vals Finance Agent overall_accuracy_pct	23.4%	32%	$4.50	SciArena LeaderboardVals Finance Agent
#6	Grok-4-0709 Strong on Vals CorpFin v2 overall_accuracy_pct and Galileo Agent Leaderboard v2 Avg TSQ	20.5%	30%	—	Vals CorpFin v2Galileo Agent Leaderboard v2
#7	gemini-3-flash-preview Strong on Vals CorpFin v2 overall_accuracy_pct and FACTS Benchmark Suite facts_grounding_score_pct	19.6%	26%	$1.13	Vals CorpFin v2FACTS Benchmark Suite
#8	gpt-5.2-2025-12-11 Strong on FACTS Benchmark Suite facts_grounding_score_pct and Vals CorpFin v2 overall_accuracy_pct	18.9%	23%	—	FACTS Benchmark SuiteVals CorpFin v2
#9	claude-sonnet-4.6 Strong on Vals Finance Agent overall_accuracy_pct and Vals CorpFin v2 overall_accuracy_pct	18.5%	24%	$6.00	Vals Finance AgentVals CorpFin v2
#10	claude-sonnet-4 Strong on Galileo Agent Leaderboard v2 Avg TSQ and Vectara HHEM Leaderboard overall_hallucination_error_pct	18.4%	29%	$6.00	Galileo Agent Leaderboard v2Vectara HHEM Leaderboard
#11	gpt-4.1-20250414 Strong on Vectara HHEM Leaderboard overall_hallucination_error_pct and Vals CorpFin v2 overall_accuracy_pct	18.3%	29%	—	Vectara HHEM LeaderboardVals CorpFin v2
#12	gemini-3.1-flash-lite-preview Strong on FACTS Benchmark Suite facts_grounding_score_pct and Vectara HHEM Leaderboard overall_hallucination_error_pct	18.0%	26%	$0.56	FACTS Benchmark SuiteVectara HHEM Leaderboard
#13	gpt-5.4-2026-03-05 Strong on Vectara HHEM Leaderboard overall_hallucination_error_pct and Vals CorpFin v2 overall_accuracy_pct	16.8%	21%	—	Vectara HHEM LeaderboardVals CorpFin v2
#14	gemini-2.5-flash Strong on FACTS Benchmark Suite facts_grounding_score_pct and Galileo Agent Leaderboard v2 Avg TSQ	16.6%	28%	$0.17	FACTS Benchmark SuiteGalileo Agent Leaderboard v2
#15	claude-opus-4-5-20251101 Strong on FACTS Benchmark Suite facts_grounding_score_pct and Vals CorpFin v2 overall_accuracy_pct	16.4%	25%	—	FACTS Benchmark SuiteVals CorpFin v2
#16	o3-20250416 Strong on SciArena Leaderboard rating_elo and Vals CorpFin v2 overall_accuracy_pct	16.1%	25%	$3.50	SciArena LeaderboardVals CorpFin v2
#17	gpt-5.1-2025-11-13 Strong on Vals Finance Agent overall_accuracy_pct and Vals CorpFin v2 overall_accuracy_pct	15.2%	24%	—	Vals Finance AgentVals CorpFin v2
#18	grok-4-fast-reasoning Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	15.1%	29%	$0.28	Vals CorpFin v2Vals Finance Agent
#19	grok-4-1-fast-reasoning Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	13.5%	20%	$0.28	Vals CorpFin v2Vals Finance Agent
#20	deepseek-r1 Strong on ContractEval Leaderboard contract_adherence_csr_pct and SYCON Bench (Table 2) sycon_unethical_tof_pct	13.5%	21%	$0.27	ContractEval LeaderboardSYCON Bench (Table 2)
#21	claude-opus-4-6-thinking Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	12.5%	14%	—	Vals CorpFin v2Vals Finance Agent
#22	grok-3 Strong on Vectara HHEM Leaderboard overall_hallucination_error_pct and Vals CorpFin v2 overall_accuracy_pct	12.2%	18%	$6.00	Vectara HHEM LeaderboardVals CorpFin v2
#23	kimi-k2.5-thinking Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	12.1%	18%	—	Vals CorpFin v2Vals Finance Agent
#24	gpt-4o Strong on LEXam Leaderboard average_score_pct and MEGA-Bench app:Planning	11.6%	16%	$0.26	LEXam LeaderboardMEGA-Bench
#25	claude-opus-4-5-20251101-thinking Strong on Vals Finance Agent overall_accuracy_pct and Vals CorpFin v2 overall_accuracy_pct	11.6%	14%	—	Vals Finance AgentVals CorpFin v2
#26	glm-5-thinking Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	11.0%	16%	—	Vals CorpFin v2Vals Finance Agent
#27	claude-sonnet-4-5-20250929-thinking Strong on Vals Finance Agent overall_accuracy_pct and Vals CorpFin v2 overall_accuracy_pct	10.8%	14%	—	Vals Finance AgentVals CorpFin v2
#29	grok-4.20-0309-reasoning Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	10.6%	14%	—	Vals CorpFin v2Vals Finance Agent
#30	qwen-2.5-72b-instruct Strong on Galileo Agent Leaderboard v2 Avg TSQ and DuckDB NSQL Leaderboard all_execution_accuracy	10.5%	18%	—	Galileo Agent Leaderboard v2DuckDB NSQL Leaderboard
#31	o4-mini Strong on Vals CorpFin v2 overall_accuracy_pct and SciArena Leaderboard rating_elo	10.4%	24%	$1.93	Vals CorpFin v2SciArena Leaderboard

Compare Models

Select two different models above to compare their evidence side by side.

▶Ranking diagnostics & missing models

Source lift

Ranked

Sources

Quality

Low

Vals CorpFin v2

44 rows · 1.2% avg lift

Vals Tax Eval v2

34 rows · 0.3% avg lift

Vals Legal Bench

33 rows · 0.3% avg lift

Vals MedQA

32 rows · 0.3% avg lift

Missing frontier models

No obvious gaps right now.

▶Taxonomy & task details

Core tasks

task.summarize_legal_contracttask.contract_term_extraction

Required modes

mode.long_contextmode.citationsmode.json_schema

Domains

domain.procurement_vendor_mgmt

Related in Supply Chain

Disruption monitoring brief

Summarize disruptions into risk, options, and recommendations.

Supplier risk monitoring

Track supplier risk signals from multi-source text and summarize actions.

Route alternatives planning

Propose alternative routes and tradeoffs under disruptions.

Tail spend categorization

Categorize tail spend purchases into taxonomy buckets for sourcing.