DevOps & SRE

Log triage

Interpret logs and propose safe diagnostic steps.

task.log_triagetask.risk_assessment

Evidence quality is currently limited for this use case. Rankings below are useful for exploration, not a strong winner claim.

Provisional leader

gpt-5-2025-08-07

Current leader based on limited benchmark evidence. Treat this ranking as directional until coverage improves.

40.9%

Best benchmark score

49.6%

Confidence

All ranked models — top 3

🥇

gpt-5-2025-08-07

40.9%

🥈

gemini-2.5-pro

37.7%

🥉

gemini-3-pro-preview

35.6%

Ranked Models

Evidence Quality

84%

Evidence Points

Top Signal

BasedAGI Log Triage Eval: overall_score_pct

All Ranked Models

30 of 30 models

Rank	Model	Score	Confidence	Price / 1M	Evidence sources
🥇	gpt-5-2025-08-07 Strong on BasedAGI Log Triage Eval overall_score_pct and SWE-bench Verified Leaderboard swe_verified_resolved_pct	40.9%	50%	—	BasedAGI Log Triage EvalSWE-bench Verified Leaderboard
🥈	gemini-2.5-pro Strong on BasedAGI Log Triage Eval overall_score_pct and FACTS Benchmark Suite facts_grounding_score_pct	37.7%	56%	$3.44	BasedAGI Log Triage EvalFACTS Benchmark Suite
🥉	gemini-3-pro-preview Strong on BasedAGI Log Triage Eval overall_score_pct and SWE-bench Verified Leaderboard swe_verified_resolved_pct	35.6%	46%	$4.50	BasedAGI Log Triage EvalSWE-bench Verified Leaderboard
#4	claude-sonnet-4.6 Strong on BasedAGI Log Triage Eval overall_score_pct and Vals Finance Agent overall_accuracy_pct	30.3%	39%	$6.00	BasedAGI Log Triage EvalVals Finance Agent
#5	gemini-3.1-pro-preview Strong on Vals Finance Agent overall_accuracy_pct and FACTS Benchmark Suite facts_search_score_pct	29.6%	34%	$4.50	Vals Finance AgentFACTS Benchmark Suite
#6	gpt-5-mini-2025-08-07 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals Finance Agent overall_accuracy_pct	28.5%	54%	—	SWE-bench Verified LeaderboardVals Finance Agent
#7	Grok-4-0709 Strong on Vals CorpFin v2 overall_accuracy_pct and Galileo Agent Leaderboard v2 Avg AC	28.3%	47%	—	Vals CorpFin v2Galileo Agent Leaderboard v2
#8	claude-sonnet-4 Strong on Galileo Agent Leaderboard v2 Avg AC and SWE-bench Verified Leaderboard swe_verified_resolved_pct	24.9%	39%	$6.00	Galileo Agent Leaderboard v2SWE-bench Verified Leaderboard
#9	gpt-5.2-2025-12-11 Strong on FACTS Benchmark Suite facts_grounding_score_pct and SWE-bench Verified Leaderboard swe_verified_resolved_pct	23.9%	28%	—	FACTS Benchmark SuiteSWE-bench Verified Leaderboard
#10	gpt-4.1-20250414 Strong on Galileo Agent Leaderboard v2 Avg AC and Vectara HHEM Leaderboard overall_hallucination_error_pct	22.7%	34%	—	Galileo Agent Leaderboard v2Vectara HHEM Leaderboard
#11	gemini-3-flash-preview Strong on Vals CorpFin v2 overall_accuracy_pct and FACTS Benchmark Suite facts_grounding_score_pct	22.6%	31%	$1.13	Vals CorpFin v2FACTS Benchmark Suite
#12	claude-opus-4-5-20251101 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and FACTS Benchmark Suite facts_grounding_score_pct	22.0%	31%	—	SWE-bench Verified LeaderboardFACTS Benchmark Suite
#13	gemini-3.1-flash-lite-preview Strong on FACTS Benchmark Suite facts_grounding_score_pct and Vectara HHEM Leaderboard overall_hallucination_error_pct	20.8%	30%	$0.56	FACTS Benchmark SuiteVectara HHEM Leaderboard
#14	o3-20250416 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals CorpFin v2 overall_accuracy_pct	20.1%	32%	$3.50	SWE-bench Verified LeaderboardVals CorpFin v2
#15	gpt-5.4-2026-03-05 Strong on Vectara HHEM Leaderboard overall_hallucination_error_pct and Vals CorpFin v2 overall_accuracy_pct	19.4%	24%	—	Vectara HHEM LeaderboardVals CorpFin v2
#16	gpt-5.1-2025-11-13 Strong on Vals Finance Agent overall_accuracy_pct and Vals CorpFin v2 overall_accuracy_pct	17.5%	27%	—	Vals Finance AgentVals CorpFin v2
#17	grok-4-fast-reasoning Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	17.4%	33%	$0.28	Vals CorpFin v2Vals Finance Agent
#18	gemini-2.5-flash Strong on FACTS Benchmark Suite facts_grounding_score_pct and Vectara HHEM Leaderboard overall_hallucination_error_pct	16.4%	28%	$0.17	FACTS Benchmark SuiteVectara HHEM Leaderboard
#19	kimi-k2.5-thinking Strong on Vals CorpFin v2 overall_accuracy_pct and OSWorld Official Leaderboard overall_score_pct	16.1%	35%	—	Vals CorpFin v2OSWorld Official Leaderboard
#20	claude-opus-4-6-thinking Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	14.4%	16%	—	Vals CorpFin v2Vals Finance Agent
#21	grok-4-1-fast-reasoning Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	14.3%	22%	$0.28	Vals CorpFin v2Vals Finance Agent
#22	grok-3 Strong on Vectara HHEM Leaderboard overall_hallucination_error_pct and Vals CorpFin v2 overall_accuracy_pct	13.4%	19%	$6.00	Vectara HHEM LeaderboardVals CorpFin v2
#23	claude-opus-4-5-20251101-thinking Strong on Vals Finance Agent overall_accuracy_pct and Vals CorpFin v2 overall_accuracy_pct	13.3%	16%	—	Vals Finance AgentVals CorpFin v2
#24	Kimi K2 Thinking Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals CorpFin v2 overall_accuracy_pct	13.1%	22%	$1.07	SWE-bench Verified LeaderboardVals CorpFin v2
#25	o4-mini Strong on Vals CorpFin v2 overall_accuracy_pct and SWE-bench Verified Leaderboard swe_verified_resolved_pct	12.8%	28%	$1.93	Vals CorpFin v2SWE-bench Verified Leaderboard
#26	claude-opus-4-6 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and AgentSet LLM Leaderboard elo_score	12.5%	16%	$10.00	SWE-bench Verified LeaderboardAgentSet LLM Leaderboard
#27	claude-sonnet-4-5-20250929-thinking Strong on Vals Finance Agent overall_accuracy_pct and Vals CorpFin v2 overall_accuracy_pct	12.5%	16%	—	Vals Finance AgentVals CorpFin v2
#29	grok-4.20-0309-reasoning Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	12.2%	16%	—	Vals CorpFin v2Vals Finance Agent
#30	glm-5-thinking Strong on Vals CorpFin v2 overall_accuracy_pct and Vals Finance Agent overall_accuracy_pct	12.2%	17%	—	Vals CorpFin v2Vals Finance Agent
#32	gpt-4.1-mini-20250414 Strong on Galileo Agent Leaderboard v2 Avg AC and Vals CorpFin v2 overall_accuracy_pct	11.5%	17%	—	Galileo Agent Leaderboard v2Vals CorpFin v2

Compare Models

Select two different models above to compare their evidence side by side.

▶Ranking diagnostics & missing models

Source lift

Ranked

Sources

Quality

Low

Vals CorpFin v2

43 rows · 1.4% avg lift

Vals Tax Eval v2

32 rows · 0.3% avg lift

Vals Finance Agent

31 rows · 1.3% avg lift

Vals MedQA

31 rows · 0.3% avg lift

Missing frontier models

No obvious gaps right now.

▶Taxonomy & task details

Core tasks

task.log_triagetask.risk_assessment

Required modes

none

Domains

domain.devops_sre

Related in DevOps & SRE

Runbook step assistant

Suggest safe runbook steps and escalation points grounded in docs.

Config debugging

Diagnose and patch YAML/JSON/TOML configs with minimal diffs.

Kubernetes manifest generation

Generate K8s manifests with safe defaults and probes.

Terraform generation

Generate Terraform IaC with correct resources and safe defaults.