Developer

Documentation from code

Generate docstrings and technical docs that match code behavior.

task.docstring_generationtask.write_report

Evidence quality is currently limited for this use case. Rankings below are useful for exploration, not a strong winner claim.

Provisional leader

gpt-5-2025-08-07

Current leader based on limited benchmark evidence. Treat this ranking as directional until coverage improves.

24.9%

Best benchmark score

29.6%

Confidence

All ranked models — top 3

🥇

gpt-5-2025-08-07

24.9%

🥈

claude-sonnet-4

22.3%

🥉

gemini-2.5-pro

19.8%

Ranked Models

Evidence Quality

80%

Evidence Points

Top Signal

Aider Polyglot Leaderboard: percent_correct_pct

All Ranked Models

30 of 30 models

Rank	Model	Score	Confidence	Price / 1M	Evidence sources
🥇	gpt-5-2025-08-07 Strong on Aider Polyglot Leaderboard percent_correct_pct and SWE-bench Verified Leaderboard swe_verified_resolved_pct	24.9%	30%	—	Aider Polyglot LeaderboardSWE-bench Verified Leaderboard
🥈	claude-sonnet-4 Strong on Galileo Agent Leaderboard v2 Avg AC and SWE-bench Verified Leaderboard swe_verified_resolved_pct	22.3%	30%	$6.00	Galileo Agent Leaderboard v2SWE-bench Verified Leaderboard
#4	gemini-2.5-pro Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and LEXam Leaderboard average_score_pct	19.8%	31%	$3.44	SWE-bench Verified LeaderboardLEXam Leaderboard
#5	gpt-4.1-20250414 Strong on Galileo Agent Leaderboard v2 Avg AC and OpenVLM OCRBench Official ocrbench_score_pct	18.8%	29%	—	Galileo Agent Leaderboard v2OpenVLM OCRBench Official
#7	gpt-4o-2024-05-13 Strong on RepoQA Official Results overall_average_pass_at_1_pct and RepoQA Official Results all_average_pass_at_1_pct	18.1%	24%	—	RepoQA Official ResultsRepoQA Official Results
#8	gpt-5-mini-2025-08-07 Strong on Vals LiveCodeBench overall_accuracy_pct and LEXam Leaderboard average_score_pct	18.1%	25%	—	Vals LiveCodeBenchLEXam Leaderboard
#9	gemini-3-pro-preview Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals SWE-bench overall_accuracy_pct	17.1%	21%	$4.50	SWE-bench Verified LeaderboardVals SWE-bench
#10	deepseek-r1 Strong on Aider Polyglot Leaderboard percent_correct_pct and LEXam Leaderboard average_score_pct	16.4%	26%	$0.27	Aider Polyglot LeaderboardLEXam Leaderboard
#11	gpt-4.1 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Aider Polyglot Leaderboard percent_correct_pct	15.7%	19%	$3.50	SWE-bench Verified LeaderboardAider Polyglot Leaderboard
#13	gpt-4o Strong on LEXam Leaderboard average_score_pct and OpenVLM OCRBench Official ocrbench_score_pct	15.1%	24%	$0.26	LEXam LeaderboardOpenVLM OCRBench Official
#14	gpt-4.1-mini-20250414 Strong on Galileo Agent Leaderboard v2 Avg AC and OpenVLM OCRBench Official ocrbench_score_pct	14.3%	23%	—	Galileo Agent Leaderboard v2OpenVLM OCRBench Official
#16	gpt-5.2-2025-12-11 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals SWE-bench overall_accuracy_pct	14.1%	16%	—	SWE-bench Verified LeaderboardVals SWE-bench
#17	o3-20250416 Strong on Aider Polyglot Leaderboard percent_correct_pct and SWE-bench Verified Leaderboard swe_verified_resolved_pct	13.8%	17%	$3.50	Aider Polyglot LeaderboardSWE-bench Verified Leaderboard
#18	gemini-3.1-pro-preview Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	13.6%	14%	$4.50	Vals SWE-benchVals LiveCodeBench
#21	gpt-4o-20241120 Strong on Aider Code Editing Leaderboard percent_correct_pct and BigCodeBench Official bigcodebench_complete_pct	13.2%	22%	—	Aider Code Editing LeaderboardBigCodeBench Official
#22	Kimi K2 Thinking Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Sonar Java Quality Leaderboard functional_skill_pct	12.9%	19%	$1.07	SWE-bench Verified LeaderboardSonar Java Quality Leaderboard
#25	qwen-2.5-72b-instruct Strong on Galileo Agent Leaderboard v2 Avg AC and Aider Code Editing Leaderboard percent_correct_pct	12.4%	17%	—	Galileo Agent Leaderboard v2Aider Code Editing Leaderboard
#28	Grok-4-0709 Strong on Vals LiveCodeBench overall_accuracy_pct and Galileo Agent Leaderboard v2 Avg AC	12.0%	18%	—	Vals LiveCodeBenchGalileo Agent Leaderboard v2
#30	gemini-3-flash-preview Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	11.4%	14%	$1.13	Vals SWE-benchVals LiveCodeBench
#31	claude-opus-4-5-20251101 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals Terminal-Bench 2 overall_accuracy_pct	11.4%	14%	—	SWE-bench Verified LeaderboardVals Terminal-Bench 2
#32	gpt-5.4-2026-03-05 Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	11.4%	13%	—	Vals SWE-benchVals LiveCodeBench
#33	claude-sonnet-4.6 Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	11.1%	13%	$6.00	Vals SWE-benchVals LiveCodeBench
#34	o4-mini Strong on Aider Polyglot Leaderboard percent_correct_pct and Vals LiveCodeBench overall_accuracy_pct	11.1%	16%	$1.93	Aider Polyglot LeaderboardVals LiveCodeBench
#35	claude-opus-4-6-thinking Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	10.9%	12%	—	Vals SWE-benchVals LiveCodeBench
#36	glm-4.7 Strong on Vals LiveCodeBench overall_accuracy_pct and Vals SWE-bench overall_accuracy_pct	10.9%	16%	—	Vals LiveCodeBenchVals SWE-bench
#37	minimax-m2.1 Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	10.9%	16%	$0.53	Vals SWE-benchVals LiveCodeBench
#40	claude-opus-4 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Aider Polyglot Leaderboard percent_correct_pct	10.7%	13%	$10.00	SWE-bench Verified LeaderboardAider Polyglot Leaderboard
#41	gpt-5.1-2025-11-13 Strong on Vals LiveCodeBench overall_accuracy_pct and Vals SWE-bench overall_accuracy_pct	10.5%	13%	—	Vals LiveCodeBenchVals SWE-bench
#42	claude-opus-4-5-20251101-thinking Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	10.5%	12%	—	Vals SWE-benchVals LiveCodeBench
#44	gemini-2.5-flash Strong on LanguageBench Grammar/Clarity Official (Split) grammar_clarity_score_pct and Galileo Agent Leaderboard v2 Avg AC	9.9%	14%	$0.17	LanguageBench Grammar/Clarity Official (Split)Galileo Agent Leaderboard v2

Compare Models

Select two different models above to compare their evidence side by side.

▶Ranking diagnostics & missing models

Source lift

Ranked

Sources

Quality

Low

Vals LiveCodeBench

38 rows · 1.1% avg lift

Vals Legal Bench

37 rows · 0.2% avg lift

Vals Tax Eval v2

35 rows · 0.2% avg lift

Vals GPQA

32 rows · 0.2% avg lift

Missing frontier models

grok-4-1-fast-non-reasoning

Thin evidence after weighting

Rank #15

14.9%

▶Taxonomy & task details

Core tasks

task.docstring_generationtask.write_report

Required modes

none

Domains

domain.software_engineering

Related in Developer

Codebase onboarding brief

Summarize a repository's architecture, modules, and conventions.

Autonomous Coding Agent

End-to-end autonomous software engineering: reading issues, writing code, running tests, submitting PRs.

Code generation

Generate correct, secure code from requirements.

Refactoring assistant

Refactor code safely while preserving behavior and improving clarity.