Engineering

Simulation setup assistant

Turn design requirements into simulation setup checklists and boundary notes.

task.planning_task_breakdowntask.write_memo_brief

Evidence quality is currently limited for this use case. Rankings below are useful for exploration, not a strong winner claim.

Provisional leader

gpt-5-2025-08-07

Current leader based on limited benchmark evidence. Treat this ranking as directional until coverage improves.

35.0%

Best benchmark score

42.7%

Confidence

All ranked models — top 3

🥇

gpt-5-2025-08-07

35.0%

🥈

claude-sonnet-4

28.0%

🥉

gemini-3-pro-preview

27.2%

Ranked Models

Evidence Quality

84%

Evidence Points

Top Signal

Aider Polyglot Leaderboard: percent_correct_pct

All Ranked Models

30 of 30 models

Rank	Model	Score	Confidence	Price / 1M	Evidence sources
🥇	gpt-5-2025-08-07 Strong on Aider Polyglot Leaderboard percent_correct_pct and SWE-bench Verified Leaderboard swe_verified_resolved_pct	35.0%	43%	—	Aider Polyglot LeaderboardSWE-bench Verified Leaderboard
🥉	claude-sonnet-4 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Galileo Agent Leaderboard v2 Avg AC	28.0%	39%	$6.00	SWE-bench Verified LeaderboardGalileo Agent Leaderboard v2
#4	gemini-3-pro-preview Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals SWE-bench overall_accuracy_pct	27.2%	33%	$4.50	SWE-bench Verified LeaderboardVals SWE-bench
#5	gpt-5.2-2025-12-11 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals SWE-bench overall_accuracy_pct	27.1%	31%	—	SWE-bench Verified LeaderboardVals SWE-bench
#6	gemini-3.1-pro-preview Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	26.2%	28%	$4.50	Vals SWE-benchVals LiveCodeBench
#7	gemini-2.5-pro Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Galileo Agent Leaderboard v2 Avg AC	25.7%	39%	$3.44	SWE-bench Verified LeaderboardGalileo Agent Leaderboard v2
#8	gpt-5-mini-2025-08-07 Strong on Vals LiveCodeBench overall_accuracy_pct and SWE-bench Verified Leaderboard swe_verified_resolved_pct	24.9%	37%	—	Vals LiveCodeBenchSWE-bench Verified Leaderboard
#9	o3-20250416 Strong on Aider Polyglot Leaderboard percent_correct_pct and SWE-bench Verified Leaderboard swe_verified_resolved_pct	24.5%	31%	$3.50	Aider Polyglot LeaderboardSWE-bench Verified Leaderboard
#10	gpt-4.1-20250414 Strong on Galileo Agent Leaderboard v2 Avg AC and SWE-bench Verified Leaderboard swe_verified_resolved_pct	22.9%	36%	—	Galileo Agent Leaderboard v2SWE-bench Verified Leaderboard
#11	Grok-4-0709 Strong on Vals LiveCodeBench overall_accuracy_pct and Galileo Agent Leaderboard v2 Avg AC	22.1%	34%	—	Vals LiveCodeBenchGalileo Agent Leaderboard v2
#12	gemini-3-flash-preview Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	22.0%	27%	$1.13	Vals SWE-benchVals LiveCodeBench
#13	claude-opus-4-5-20251101 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals Terminal-Bench 2 overall_accuracy_pct	22.0%	26%	—	SWE-bench Verified LeaderboardVals Terminal-Bench 2
#14	gpt-5.4-2026-03-05 Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	21.9%	25%	—	Vals SWE-benchVals LiveCodeBench
#15	claude-sonnet-4.6 Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	21.4%	25%	$6.00	Vals SWE-benchVals LiveCodeBench
#16	claude-opus-4-6-thinking Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	21.0%	23%	—	Vals SWE-benchVals LiveCodeBench
#17	gpt-5.1-2025-11-13 Strong on Vals LiveCodeBench overall_accuracy_pct and Vals SWE-bench overall_accuracy_pct	20.2%	25%	—	Vals LiveCodeBenchVals SWE-bench
#18	claude-opus-4-5-20251101-thinking Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	20.2%	23%	—	Vals SWE-benchVals LiveCodeBench
#19	gpt-4o-20241120 Strong on Aider Code Editing Leaderboard percent_correct_pct and BigCodeBench Official bigcodebench_complete_pct	20.1%	36%	—	Aider Code Editing LeaderboardBigCodeBench Official
#20	o4-mini Strong on Aider Polyglot Leaderboard percent_correct_pct and Vals LiveCodeBench overall_accuracy_pct	19.6%	29%	$1.93	Aider Polyglot LeaderboardVals LiveCodeBench
#21	Kimi K2 Thinking Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals SWE-bench overall_accuracy_pct	19.2%	29%	$1.07	SWE-bench Verified LeaderboardVals SWE-bench
#22	kimi-k2.5-thinking Strong on Vals LiveCodeBench overall_accuracy_pct and Vals SWE-bench overall_accuracy_pct	18.3%	24%	—	Vals LiveCodeBenchVals SWE-bench
#23	gpt-4o-2024-05-13 Strong on Aider Code Editing Leaderboard percent_correct_pct and BigCodeBench Official bigcodebench_complete_pct	18.3%	28%	—	Aider Code Editing LeaderboardBigCodeBench Official
#24	claude-opus-4 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Aider Polyglot Leaderboard percent_correct_pct	18.2%	23%	$10.00	SWE-bench Verified LeaderboardAider Polyglot Leaderboard
#25	qwen-2.5-72b-instruct Strong on Galileo Agent Leaderboard v2 Avg AC and Aider Code Editing Leaderboard percent_correct_pct	18.2%	25%	—	Galileo Agent Leaderboard v2Aider Code Editing Leaderboard
#27	claude-sonnet-4-5-20250929-thinking Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	17.5%	23%	—	Vals SWE-benchVals LiveCodeBench
#28	gemini-3.1-flash-lite-preview Strong on Vals LiveCodeBench overall_accuracy_pct and Vals SWE-bench overall_accuracy_pct	17.2%	26%	$0.56	Vals LiveCodeBenchVals SWE-bench
#29	grok-4.20-0309-reasoning Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	16.8%	23%	—	Vals SWE-benchVals LiveCodeBench
#30	minimax-m2.1 Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	16.6%	23%	$0.53	Vals SWE-benchVals LiveCodeBench
#31	glm-5-thinking Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	16.5%	21%	—	Vals SWE-benchVals LiveCodeBench
#32	MiniMax-M2.7 Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	16.4%	21%	$0.53	Vals SWE-benchVals LiveCodeBench

Compare Models

Select two different models above to compare their evidence side by side.

▶Ranking diagnostics & missing models

Source lift

Ranked

Sources

Quality

Low

Vals LiveCodeBench

44 rows · 1.8% avg lift

Vals Legal Bench

44 rows · 0.4% avg lift

Vals Tax Eval v2

42 rows · 0.4% avg lift

Vals MedQA

40 rows · 0.4% avg lift

Missing frontier models

No obvious gaps right now.

▶Taxonomy & task details

Core tasks

task.planning_task_breakdowntask.write_memo_brief

Required modes

none

Domains

domain.mechanical_engineeringdomain.civil_engineering

Related in Engineering

Component selection assistant

Recommend components under constraints with evidence and tradeoffs.

🥇gemini-3.1-pro-preview

35.5%

CAD scripting helper

Generate and debug CAD automation scripts and parametric geometry code.

Verilog/VHDL generation

Generate RTL code and testbenches from functional specs.