Engineering

Verilog/VHDL generation

Generate RTL code and testbenches from functional specs.

task.code_generationtask.test_generation_unit

Evidence quality is currently limited for this use case. Rankings below are useful for exploration, not a strong winner claim.

Provisional leader

claude-sonnet-4

Current leader based on limited benchmark evidence. Treat this ranking as directional until coverage improves.

32.7%

Best benchmark score

45.5%

Confidence

All ranked models — top 3

🥇

claude-sonnet-4

32.7%

🥈

gpt-5-2025-08-07

31.0%

🥉

Kimi K2 Thinking

25.4%

Ranked Models

Evidence Quality

81%

Evidence Points

Top Signal

SWE-bench Verified Leaderboard: swe_verified_resolved_pct

All Ranked Models

30 of 30 models

Rank	Model	Score	Confidence	Price / 1M	Evidence sources
🥇	claude-sonnet-4 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Aider Polyglot Leaderboard percent_correct_pct	32.7%	46%	$6.00	SWE-bench Verified LeaderboardAider Polyglot Leaderboard
🥈	gpt-5-2025-08-07 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Aider Polyglot Leaderboard percent_correct_pct	31.0%	36%	—	SWE-bench Verified LeaderboardAider Polyglot Leaderboard
#4	Kimi K2 Thinking Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Sonar Java Quality Leaderboard functional_skill_pct	25.4%	35%	$1.07	SWE-bench Verified LeaderboardSonar Java Quality Leaderboard
#5	o3-20250416 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Aider Polyglot Leaderboard percent_correct_pct	23.6%	31%	$3.50	SWE-bench Verified LeaderboardAider Polyglot Leaderboard
#6	gemini-3-pro-preview Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals SWE-bench overall_accuracy_pct	23.5%	28%	$4.50	SWE-bench Verified LeaderboardVals SWE-bench
#7	gpt-5.2-2025-12-11 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals SWE-bench overall_accuracy_pct	23.0%	27%	—	SWE-bench Verified LeaderboardVals SWE-bench
#8	gpt-4o-2024-05-13 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and RepoQA Official Results overall_average_pass_at_1_pct	22.7%	33%	—	SWE-bench Verified LeaderboardRepoQA Official Results
#9	gemini-2.5-pro Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Galileo Agent Leaderboard v2 Avg AC	21.7%	30%	$3.44	SWE-bench Verified LeaderboardGalileo Agent Leaderboard v2
#10	gpt-4o-20241120 Strong on BigCodeBench Official bigcodebench_complete_pct and BigCodeBench Official bigcodebench_instruct_pct	21.4%	40%	—	BigCodeBench OfficialBigCodeBench Official
#11	deepseek-r1 Strong on Aider Polyglot Leaderboard percent_correct_pct and Sonar Java Quality Leaderboard functional_skill_pct	20.4%	28%	$0.27	Aider Polyglot LeaderboardSonar Java Quality Leaderboard
#12	claude-opus-4-5-20251101 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals Terminal-Bench 2 overall_accuracy_pct	20.3%	24%	—	SWE-bench Verified LeaderboardVals Terminal-Bench 2
#13	claude-opus-4 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Aider Polyglot Leaderboard percent_correct_pct	20.3%	25%	$10.00	SWE-bench Verified LeaderboardAider Polyglot Leaderboard
#14	gpt-4.1-20250414 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Galileo Agent Leaderboard v2 Avg AC	19.8%	33%	—	SWE-bench Verified LeaderboardGalileo Agent Leaderboard v2
#15	gpt-4.1 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Aider Polyglot Leaderboard percent_correct_pct	19.6%	22%	$3.50	SWE-bench Verified LeaderboardAider Polyglot Leaderboard
#16	gpt-5-mini-2025-08-07 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Vals LiveCodeBench overall_accuracy_pct	19.6%	29%	—	SWE-bench Verified LeaderboardVals LiveCodeBench
#17	o4-mini Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Aider Polyglot Leaderboard percent_correct_pct	19.1%	30%	$1.93	SWE-bench Verified LeaderboardAider Polyglot Leaderboard
#21	glm-4.7 Strong on Sonar Java Quality Leaderboard functional_skill_pct and Vals LiveCodeBench overall_accuracy_pct	17.1%	25%	—	Sonar Java Quality LeaderboardVals LiveCodeBench
#22	minimax-m2.1 Strong on Sonar Java Quality Leaderboard functional_skill_pct and Vals SWE-bench overall_accuracy_pct	16.4%	25%	$0.53	Sonar Java Quality LeaderboardVals SWE-bench
#24	gemini-3.1-pro-preview Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	15.4%	16%	$4.50	Vals SWE-benchVals LiveCodeBench
#25	qwen-2.5-72b-instruct Strong on BigCodeBench Official bigcodebench_complete_pct and BigCodeBench Official bigcodebench_instruct_pct	15.0%	20%	—	BigCodeBench OfficialBigCodeBench Official
#33	Grok-4-0709 Strong on Vals LiveCodeBench overall_accuracy_pct and Galileo Agent Leaderboard v2 Avg AC	13.8%	21%	—	Vals LiveCodeBenchGalileo Agent Leaderboard v2
#34	deepseek-v3 Strong on BigCodeBench Official bigcodebench_complete_pct and BigCodeBench Official bigcodebench_instruct_pct	13.8%	17%	—	BigCodeBench OfficialBigCodeBench Official
#35	gemini-3-flash-preview Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	13.0%	16%	$1.13	Vals SWE-benchVals LiveCodeBench
#36	gpt-4o Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and Sonar Java Quality Leaderboard functional_skill_pct	13.0%	30%	$0.26	SWE-bench Verified LeaderboardSonar Java Quality Leaderboard
#37	gpt-5.4-2026-03-05 Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	12.9%	14%	—	Vals SWE-benchVals LiveCodeBench
#38	gpt-4o-mini-2024-07-18 Strong on BigCodeBench Official bigcodebench_complete_pct and BigCodeBench Official bigcodebench_instruct_pct	12.9%	21%	—	BigCodeBench OfficialBigCodeBench Official
#40	claude-opus-4-6 Strong on SWE-bench Verified Leaderboard swe_verified_resolved_pct and AgentSet LLM Leaderboard elo_score	12.7%	14%	$10.00	SWE-bench Verified LeaderboardAgentSet LLM Leaderboard
#41	claude-sonnet-4.6 Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	12.6%	14%	$6.00	Vals SWE-benchVals LiveCodeBench
#43	GLM-5 Strong on Sonar Java Quality Leaderboard functional_skill_pct and Sonar Java Quality Leaderboard issue_density_error_per_kloc	12.4%	14%	—	Sonar Java Quality LeaderboardSonar Java Quality Leaderboard
#45	claude-opus-4-6-thinking Strong on Vals SWE-bench overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	12.3%	13%	—	Vals SWE-benchVals LiveCodeBench

Compare Models

Select two different models above to compare their evidence side by side.

▶Ranking diagnostics & missing models

Source lift

Ranked

Sources

Quality

Low

Vals LiveCodeBench

39 rows · 1.1% avg lift

Vals Legal Bench

38 rows · 0.2% avg lift

Vals Tax Eval v2

36 rows · 0.2% avg lift

Vals MedQA

34 rows · 0.2% avg lift

Missing frontier models

No obvious gaps right now.

▶Taxonomy & task details

Core tasks

task.code_generationtask.test_generation_unit

Required modes

none

Domains

domain.eda_verilog_vhdl

Related in Engineering

Component selection assistant

Recommend components under constraints with evidence and tradeoffs.

🥇gemini-3.1-pro-preview

35.5%

Simulation setup assistant

Turn design requirements into simulation setup checklists and boundary notes.

CAD scripting helper

Generate and debug CAD automation scripts and parametric geometry code.