Data & Analytics

Metric definition workshop

Turn ambiguous KPI definitions into precise, measurable specs.

task.metric_definition_clarificationtask.decision_recommendation

Evidence quality is currently limited for this use case. Rankings below are useful for exploration, not a strong winner claim.

Provisional leader

gpt-4o

Current leader based on limited benchmark evidence. Treat this ranking as directional until coverage improves.

30.3%

Best benchmark score

54.8%

Confidence

All ranked models — top 3

🥇

gpt-4o

30.3%

🥈

deepseek-r1

27.5%

🥉

qwen-2.5-72b-instruct

24.9%

Ranked Models

Evidence Quality

82%

Evidence Points

Top Signal

DuckDB NSQL Leaderboard: all_execution_accuracy

All Ranked Models

30 of 30 models

Rank	Model	Score	Confidence	Price / 1M	Evidence sources
🥇	gpt-4o Strong on DuckDB NSQL Leaderboard all_execution_accuracy and JSONSchemaBench Leaderboard medium_schema_compliance_pct	30.3%	55%	$0.26	DuckDB NSQL LeaderboardJSONSchemaBench Leaderboard
🥈	deepseek-r1 Strong on DuckDB NSQL Leaderboard all_execution_accuracy and LiveSQLBench success_rate_pct	27.5%	47%	$0.27	DuckDB NSQL LeaderboardLiveSQLBench
#4	qwen-2.5-72b-instruct Strong on DuckDB NSQL Leaderboard all_execution_accuracy and JSONSchemaBench Leaderboard medium_schema_compliance_pct	24.9%	35%	—	DuckDB NSQL LeaderboardJSONSchemaBench Leaderboard
#6	gpt-4o-20241120 Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	23.6%	40%	—	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#7	gpt-5-2025-08-07 Strong on LiveSQLBench success_rate_pct and Spider2.0 Snow Text-to-SQL snow_text_to_sql_score_pct	21.2%	29%	—	LiveSQLBenchSpider2.0 Snow Text-to-SQL
#8	o3-20250416 Strong on LiveSQLBench success_rate_pct and Spider2.0 Snow Text-to-SQL snow_text_to_sql_score_pct	20.8%	31%	$3.50	LiveSQLBenchSpider2.0 Snow Text-to-SQL
#10	claude-sonnet-4 Strong on LiveSQLBench success_rate_pct and Spider2.0 Lite Text-to-SQL lite_text_to_sql_score_pct	20.1%	36%	$6.00	LiveSQLBenchSpider2.0 Lite Text-to-SQL
#12	gpt-4.1 Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	17.8%	24%	$3.50	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#15	gpt-4o-mini-2024-07-18 Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	16.9%	27%	—	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#16	Claude-3.5-Sonnet Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	16.5%	26%	$6.00	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#18	gpt-4o-2024-08-06 Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	15.8%	25%	—	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#25	Llama-3.3-70B-Instruct Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	13.6%	20%	—	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#29	gemini-2.0-flash-001 Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	13.1%	24%	—	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#30	Qwen3-30B-A3B Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	12.9%	21%	—	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#32	Qwen2.5-Coder-7B Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	12.4%	19%	—	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#33	gemma-2-27b-it Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	12.0%	21%	—	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#35	phi-4 Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	11.6%	21%	—	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#36	o4-mini Strong on LiveSQLBench success_rate_pct and Aider Polyglot Leaderboard percent_correct_pct	11.6%	18%	$1.93	LiveSQLBenchAider Polyglot Leaderboard
#38	Qwen3-32B Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	11.5%	20%	—	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#39	deepseek-v3 Strong on LiveSQLBench success_rate_pct and Spider2.0 Lite Text-to-SQL lite_text_to_sql_score_pct	11.4%	30%	—	LiveSQLBenchSpider2.0 Lite Text-to-SQL
#41	claude-opus-4-6 Strong on LiveSQLBench success_rate_pct and AgentSet LLM Leaderboard elo_score	11.2%	12%	$10.00	LiveSQLBenchAgentSet LLM Leaderboard
#45	QwQ-32B-Preview Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	10.8%	19%	—	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#46	Phi-3-medium-128k-instruct Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	10.4%	19%	—	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#47	Qwen2.5-32B-Instruct Strong on JSONSchemaBench Leaderboard medium_schema_compliance_pct and JSONSchemaBench Leaderboard hard_schema_compliance_pct	10.4%	11%	—	JSONSchemaBench LeaderboardJSONSchemaBench Leaderboard
#49	gemini-2.5-pro Strong on Galileo Agent Leaderboard v2 Avg AC and Galileo Agent Leaderboard v2 Avg TSQ	10.0%	16%	$3.44	Galileo Agent Leaderboard v2Galileo Agent Leaderboard v2
#50	gemini-3.1-pro-preview Strong on Vals LiveCodeBench overall_accuracy_pct and Vals SWE-bench overall_accuracy_pct	9.8%	11%	$4.50	Vals LiveCodeBenchVals SWE-bench
#51	minimax-m2.1 Strong on LiveSQLBench success_rate_pct and Vals SWE-bench overall_accuracy_pct	9.5%	16%	$0.53	LiveSQLBenchVals SWE-bench
#53	Meta-Llama-3.1-8B Strong on DuckDB NSQL Leaderboard all_execution_accuracy and DuckDB NSQL Leaderboard hard_execution_accuracy	9.1%	19%	—	DuckDB NSQL LeaderboardDuckDB NSQL Leaderboard
#54	Grok-4-0709 Strong on Galileo Agent Leaderboard v2 Avg TSQ and Galileo Agent Leaderboard v2 Avg AC	9.0%	13%	—	Galileo Agent Leaderboard v2Galileo Agent Leaderboard v2
#56	gpt-5-mini-2025-08-07 Strong on Vals MedQA overall_accuracy_pct and Vals LiveCodeBench overall_accuracy_pct	8.8%	13%	—	Vals MedQAVals LiveCodeBench

Compare Models

Select two different models above to compare their evidence side by side.

▶Ranking diagnostics & missing models

Source lift

Ranked

Sources

Quality

Low

DuckDB NSQL Leaderboard

24 rows · 4.1% avg lift

EQ-Bench Leaderboard

21 rows · 0.4% avg lift

Vals Legal Bench

17 rows · 0.4% avg lift

Vals MedQA

15 rows · 0.4% avg lift

Missing frontier models

gpt-5.2-2025-12-11

Thin evidence after weighting

Rank #9

21.9%

claude-opus-4-5-20251101

Thin evidence after weighting

Rank #10

18.6%

claude-sonnet-4.6

Thin evidence after weighting

Rank #11

20.0%

grok-4-1-fast-reasoning

Thin evidence after weighting

Rank #12

19.5%

▶Taxonomy & task details

Core tasks

task.metric_definition_clarificationtask.decision_recommendation

Required modes

none

Domains

domain.data_analytics_bi

Related in Data & Analytics

SQL debugging

Diagnose and fix SQL queries for correctness and performance.

Dashboard narratives

Generate weekly KPI narratives and investigation suggestions.

Chart & Data Visualization Interpretation

Reading charts, graphs, and dashboards to extract insights and answer questions.

Text-to-SQL analyst assistant

Convert questions into SQL and explain the query.