Developer

IDE code completion

Fast local-context code completion and small snippet generation.

task.code_completiontask.api_usage_correctness

Evidence quality is currently limited for this use case. Rankings below are useful for exploration, not a strong winner claim.

Provisional leader

claude-opus-4-6

Current leader based on limited benchmark evidence. Treat this ranking as directional until coverage improves.

27.6%

Best benchmark score

29.9%

Confidence

All ranked models — top 3

🥇

claude-opus-4-6

27.6%

🥈

gpt-5-2025-08-07

26.9%

🥉

claude-sonnet-4

22.8%

Ranked Models

Evidence Quality

82%

Evidence Points

Top Signal

SWE-bench Verified Leaderboard: swe_verified_resolved_pct

All Ranked Models

30 of 30 models

Rank	Model	Score	Confidence	Price / 1M	Evidence sources
🥇	claude-opus-4-6 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and OpenHands Index average_score_pct	27.6%	30%	$10.00	SWE-bench Verified LeaderboardOpenHands Index
🥈	gpt-5-2025-08-07 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Aider Polyglot Leaderboard percent_correct_pct	26.9%	31%	—	SWE-bench Verified LeaderboardAider Polyglot Leaderboard
#5	claude-sonnet-4 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Aider Polyglot Leaderboard percent_correct_pct	22.8%	31%	$6.00	SWE-bench Verified LeaderboardAider Polyglot Leaderboard
#9	claude-sonnet-4.6 Strong on OpenHands Index greenfield_score_pct and OpenHands Issue Resolution issue_resolution_score_pct	18.5%	33%	$6.00	OpenHands IndexOpenHands Issue Resolution
#10	Kimi K2 Thinking Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Sonar Java Quality Leaderboard functional_skill_pct	18.1%	44%	$1.07	SWE-bench Verified LeaderboardSonar Java Quality Leaderboard
#11	GLM-5 Strong on Sonar Java Quality Leaderboard functional_skill_pct and Sonar Java Quality Leaderboard issue_density_error_per_kloc	17.6%	31%	—	Sonar Java Quality LeaderboardSonar Java Quality Leaderboard
#13	gpt-4o-2024-05-13 Strong on BigCodeBench Official bigcodebench_complete_pct and BigCodeBench Official bigcodebench_instruct_pct	16.4%	23%	—	BigCodeBench OfficialBigCodeBench Official
#14	gpt-4o-20241120 Strong on BigCodeBench Official bigcodebench_complete_pct and BigCodeBench Official bigcodebench_instruct_pct	16.3%	27%	—	BigCodeBench OfficialBigCodeBench Official
#16	o3-20250416 Strong on Aider Polyglot Leaderboard percent_correct_pct and SWE-bench Verified Leaderboard swe_verified_resolved_pct	16.0%	20%	$3.50	Aider Polyglot LeaderboardSWE-bench Verified Leaderboard
#18	kimi-k2.5-thinking Strong on Vals LiveCodeBench overall_accuracy_pct and Vals SWE-bench overall_accuracy_pct	15.5%	33%	—	Vals LiveCodeBenchVals SWE-bench
#20	deepseek-r1 Strong on Aider Polyglot Leaderboard percent_correct_pct and Sonar Java Quality Leaderboard functional_skill_pct	14.8%	20%	$0.27	Aider Polyglot LeaderboardSonar Java Quality Leaderboard
#21	gemini-3-pro-preview Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals SWE-bench overall_accuracy_pct	14.7%	18%	$4.50	SWE-bench Verified LeaderboardVals SWE-bench
#23	minimax-m2.1 Strong on Sonar Java Quality Leaderboard functional_skill_pct and Vals SWE-bench overall_accuracy_pct	14.5%	39%	$0.53	Sonar Java Quality LeaderboardVals SWE-bench
#24	gemini-2.5-pro Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Galileo Agent Leaderboard v2 Avg AC	13.9%	20%	$3.44	SWE-bench Verified LeaderboardGalileo Agent Leaderboard v2
#25	gpt-5.2-2025-12-11 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals SWE-bench overall_accuracy_pct	13.8%	16%	—	SWE-bench Verified LeaderboardVals SWE-bench
#27	claude-opus-4 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Aider Polyglot Leaderboard percent_correct_pct	13.6%	17%	$10.00	SWE-bench Verified LeaderboardAider Polyglot Leaderboard
#28	qwen-2.5-72b-instruct Strong on BigCodeBench Official bigcodebench_complete_pct and Galileo Agent Leaderboard v2 Avg AC	13.4%	18%	—	BigCodeBench OfficialGalileo Agent Leaderboard v2
#31	o4-mini Strong on Aider Polyglot Leaderboard percent_correct_pct and SWE-bench Verified Leaderboard swe_verified_resolved_pct	12.9%	19%	$1.93	Aider Polyglot LeaderboardSWE-bench Verified Leaderboard
#32	gpt-4.1 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Aider Polyglot Leaderboard percent_correct_pct	12.9%	15%	$3.50	SWE-bench Verified LeaderboardAider Polyglot Leaderboard
#34	gpt-4.1-20250414 Strong on Galileo Agent Leaderboard v2 Avg AC and SWE-bench Verified Leaderboard swe_verified_resolved_pct	12.7%	20%	—	Galileo Agent Leaderboard v2SWE-bench Verified Leaderboard
#35	gpt-5-mini-2025-08-07 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals LiveCodeBench overall_accuracy_pct	12.5%	18%	—	SWE-bench Verified LeaderboardVals LiveCodeBench
#38	claude-opus-4-5-20251101 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals Terminal-Bench 2 overall_accuracy_pct	12.0%	14%	—	SWE-bench Verified LeaderboardVals Terminal-Bench 2
#40	glm-4.7 Strong on Sonar Java Quality Leaderboard functional_skill_pct and Vals LiveCodeBench overall_accuracy_pct	11.3%	17%	—	Sonar Java Quality LeaderboardVals LiveCodeBench
#41	gpt-4o-2024-08-06 Strong on BigCodeBench Official bigcodebench_hard_complete_pct and Aider Code Editing Leaderboard percent_correct_pct	11.3%	21%	—	BigCodeBench OfficialAider Code Editing Leaderboard
#42	gemini-3.1-pro-preview Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	11.2%	12%	$4.50	Vals SWE-benchVals LiveCodeBench
#44	deepseek-v3 Strong on BigCodeBench Official bigcodebench_complete_pct and BigCodeBench Official bigcodebench_instruct_pct	10.7%	13%	—	BigCodeBench OfficialBigCodeBench Official
#48	gpt-4o Strong on Sonar Java Quality Leaderboard functional_skill_pct and SWE-bench Verified Leaderboard swe_verified_resolved_pct	10.0%	20%	$0.26	Sonar Java Quality LeaderboardSWE-bench Verified Leaderboard
#49	Grok-4-0709 Strong on Vals LiveCodeBench overall_accuracy_pct and Galileo Agent Leaderboard v2 Avg AC	9.9%	15%	—	Vals LiveCodeBenchGalileo Agent Leaderboard v2
#50	gpt-4o-mini-2024-07-18 Strong on BigCodeBench Official bigcodebench_complete_pct and BigCodeBench Official bigcodebench_instruct_pct	9.8%	18%	—	BigCodeBench OfficialBigCodeBench Official
#51	gemini-3-flash-preview Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	9.5%	11%	$1.13	Vals SWE-benchVals LiveCodeBench

Compare Models

Select two different models above to compare their evidence side by side.

▶Ranking diagnostics & missing models

Source lift

Ranked

Sources

Quality

Low

Vals LiveCodeBench

27 rows · 1.2% avg lift

Vals Legal Bench

24 rows · 0.3% avg lift

Vals MedQA

23 rows · 0.3% avg lift

Vals Tax Eval v2

21 rows · 0.3% avg lift

Missing frontier models

grok-4-1-fast-non-reasoning

Thin evidence after weighting

Rank #15

14.9%

▶Taxonomy & task details

Core tasks

task.code_completiontask.api_usage_correctness

Required modes

none

Domains

domain.software_engineering

Related in Developer

Codebase onboarding brief

Summarize a repository's architecture, modules, and conventions.

Autonomous Coding Agent

End-to-end autonomous software engineering: reading issues, writing code, running tests, submitting PRs.

Code generation

Generate correct, secure code from requirements.

Refactoring assistant

Refactor code safely while preserving behavior and improving clarity.