Risk & Eval

Disinformation and manipulation resistance (eval)

Measure refusal and safe handling of deceptive content generation requests.

task.refusal_rate_by_categorytask.jailbreak_resistance

Evidence quality is currently limited for this use case. Rankings below are useful for exploration, not a strong winner claim.

Provisional leader

gemini-3.1-pro-preview

Current leader based on limited benchmark evidence. Treat this ranking as directional until coverage improves.

33.2%

Best benchmark score

38.1%

Confidence

All ranked models — top 3

🥇

gemini-3.1-pro-preview

33.2%

🥈

gemini-2.5-pro

28.0%

🥉

gpt-5-2025-08-07

27.3%

Ranked Models

Evidence Quality

83%

Evidence Points

Top Signal

Vals Finance Agent: overall_accuracy_pct

All Ranked Models

30 of 30 models

Rank	Model	Score	Confidence	Price / 1M	Evidence sources
🥇	gemini-3.1-pro-preview Strong on Vals Finance Agent overall_accuracy_pct and FACTS Benchmark Suite facts_search_score_pct	33.2%	38%	$4.50	Vals Finance AgentFACTS Benchmark Suite
🥈	gemini-2.5-pro Strong on FACTS Benchmark Suite facts_grounding_score_pct and Vectara HHEM Leaderboard overall_hallucination_error_pct	28.0%	41%	$3.44	FACTS Benchmark SuiteVectara HHEM Leaderboard
🥉	gpt-5-2025-08-07 Strong on FACTS Benchmark Suite facts_grounding_score_pct and Vals Finance Agent overall_accuracy_pct	27.3%	37%	—	FACTS Benchmark SuiteVals Finance Agent
#4	gemini-3-flash-preview Strong on Vals CorpFin v2 overall_accuracy_pct and FACTS Benchmark Suite facts_grounding_score_pct	26.2%	35%	$1.13	Vals CorpFin v2FACTS Benchmark Suite
#5	gpt-5-mini-2025-08-07 Strong on Vals Finance Agent overall_accuracy_pct and Vals CorpFin v2 overall_accuracy_pct	25.2%	40%	—	Vals Finance AgentVals CorpFin v2
#6	Grok-4-0709 Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	24.3%	35%	—	Vals CorpFin v2Vals Finance Agent
#7	claude-sonnet-4.6 Strong on Vals Finance Agent overall_accuracy_pct and Vals CorpFin v2 overall_accuracy_pct	23.9%	30%	$6.00	Vals Finance AgentVals CorpFin v2
#8	gemini-3-pro-preview Strong on Vals Finance Agent overall_accuracy_pct and Vals CorpFin v2 overall_accuracy_pct	23.3%	33%	$4.50	Vals Finance AgentVals CorpFin v2
#9	gpt-5.2-2025-12-11 Strong on FACTS Benchmark Suite facts_grounding_score_pct and Vals CorpFin v2 overall_accuracy_pct	22.5%	29%	—	FACTS Benchmark SuiteVals CorpFin v2
#10	gemini-3.1-flash-lite-preview Strong on FACTS Benchmark Suite facts_grounding_score_pct and Vectara HHEM Leaderboard overall_hallucination_error_pct	22.4%	33%	$0.56	FACTS Benchmark SuiteVectara HHEM Leaderboard
#11	gpt-5.4-2026-03-05 Strong on Vectara HHEM Leaderboard overall_hallucination_error_pct and Vals CorpFin v2 overall_accuracy_pct	21.4%	28%	—	Vectara HHEM LeaderboardVals CorpFin v2
#12	gpt-4.1-20250414 Strong on Vectara HHEM Leaderboard overall_hallucination_error_pct and Vals CorpFin v2 overall_accuracy_pct	21.0%	31%	—	Vectara HHEM LeaderboardVals CorpFin v2
#13	gpt-5.1-2025-11-13 Strong on Vals Finance Agent overall_accuracy_pct and Vals CorpFin v2 overall_accuracy_pct	20.5%	31%	—	Vals Finance AgentVals CorpFin v2
#14	claude-opus-4-5-20251101 Strong on FACTS Benchmark Suite facts_grounding_score_pct and Vals CorpFin v2 overall_accuracy_pct	20.0%	31%	—	FACTS Benchmark SuiteVals CorpFin v2
#15	grok-4-fast-reasoning Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	19.9%	38%	$0.28	Vals CorpFin v2Vals Finance Agent
#16	claude-sonnet-4 Strong on Vectara HHEM Leaderboard overall_hallucination_error_pct and FACTS Benchmark Suite facts_grounding_score_pct	19.8%	34%	$6.00	Vectara HHEM LeaderboardFACTS Benchmark Suite
#17	o3-20250416 Strong on Vals CorpFin v2 overall_accuracy_pct and SciArena Leaderboard rating_elo	19.3%	30%	$3.50	Vals CorpFin v2SciArena Leaderboard
#18	grok-4-1-fast-reasoning Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	16.9%	25%	$0.28	Vals CorpFin v2Vals Finance Agent
#19	gemini-2.5-flash Strong on FACTS Benchmark Suite facts_grounding_score_pct and Vectara HHEM Leaderboard overall_hallucination_error_pct	16.3%	27%	$0.17	FACTS Benchmark SuiteVectara HHEM Leaderboard
#20	kimi-k2.5-thinking Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	15.9%	22%	—	Vals CorpFin v2Vals Finance Agent
#21	grok-3 Strong on Vectara HHEM Leaderboard overall_hallucination_error_pct and Vals CorpFin v2 overall_accuracy_pct	15.9%	22%	$6.00	Vectara HHEM LeaderboardVals CorpFin v2
#22	claude-opus-4-6-thinking Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	15.5%	17%	—	Vals CorpFin v2Vals Finance Agent
#23	claude-opus-4-5-20251101-thinking Strong on Vals Finance Agent overall_accuracy_pct and Vals CorpFin v2 overall_accuracy_pct	14.4%	17%	—	Vals Finance AgentVals CorpFin v2
#24	claude-sonnet-4-5-20250929-thinking Strong on Vals Finance Agent overall_accuracy_pct and Vals CorpFin v2 overall_accuracy_pct	13.5%	17%	—	Vals Finance AgentVals CorpFin v2
#26	grok-4.20-0309-reasoning Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	13.2%	17%	—	Vals CorpFin v2Vals Finance Agent
#27	glm-5-thinking Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	13.1%	19%	—	Vals CorpFin v2Vals Finance Agent
#28	gpt-4o-2024-05-13 Strong on LLM Trustworthy Leaderboard privacy and LLM Trustworthy Leaderboard adv	12.9%	24%	—	LLM Trustworthy LeaderboardLLM Trustworthy Leaderboard
#29	o4-mini Strong on Vals CorpFin v2 overall_accuracy_pct and Vals CorpFin v2 shared_max_context_accuracy_pct	12.9%	29%	$1.93	Vals CorpFin v2Vals CorpFin v2
#30	Llama-2-7b-chat-hf Strong on LLM Trustworthy Leaderboard fairness and LLM Trustworthy Leaderboard privacy	12.7%	16%	—	LLM Trustworthy LeaderboardLLM Trustworthy Leaderboard
#31	Kimi K2 Thinking Strong on Vals CorpFin v2 overall_accuracy_pct and Vals CorpFin v2 shared_max_context_accuracy_pct	12.6%	21%	$1.07	Vals CorpFin v2Vals CorpFin v2

Compare Models

Select two different models above to compare their evidence side by side.

▶Ranking diagnostics & missing models

Source lift

Ranked

Sources

Quality

Low

Vals CorpFin v2

44 rows · 1.4% avg lift

Vals MedQA

35 rows · 0.3% avg lift

Vals Tax Eval v2

34 rows · 0.3% avg lift

Vals Legal Bench

33 rows · 0.4% avg lift

Missing frontier models

No obvious gaps right now.

▶Taxonomy & task details

Core tasks

task.refusal_rate_by_categorytask.jailbreak_resistance

Required modes

none

Domains

domain.general_businessdomain.geopolitics_intel

Related in Risk & Eval

Crisis escalation protocol (eval)

Measure safe crisis escalation behavior under the selected policy.

Jailbreak resistance (eval)

Measure robustness to adversarial prompts that attempt to bypass policy.

Overrefusal (eval)

Measure how often benign requests are incorrectly refused.

Refusal profile (eval)

Measure refusal/overrefusal rates across predefined categories.