Noometry

Model comparison

o1-mini vs Qwen2.5-Coder-32B

o1-mini and Qwen2.5-Coder-32B score almost the same on the Noometry Index (34.0 vs 33.4), so choose on price, context window or the category you care about most.

Last verified . 23 shared benchmarks.

o1-mini OpenAI

34.0

Rank #235 Confirmed

Qwen2.5-Coder-32B Alibaba (Qwen)

33.4

Rank #245 Confirmed

Summary

  • They share 23 benchmarks with published results for both. o1-mini scores higher in 7 categories and Qwen2.5-Coder-32B in 1 category; 8 gaps are clear of the uncertainty.
  • The widest gap is in coding, where o1-mini leads 35.5 to 22.6.
  • The biggest single-benchmark swing is LiveBench Reasoning: 72.3% for o1-mini and 42.1% for Qwen2.5-Coder-32B.
  • Qwen2.5-Coder-32B has downloadable open weights; the other is API-only.

Side by side

o1-miniQwen2.5-Coder-32B
ProviderOpenAIAlibaba (Qwen)
Noometry Index34.033.4
Released2024-09-122024-09-18
WeightsProprietaryOpen
Context window—33K
Max output—29K
Input $ / M tokens—$0.66
Output $ / M tokens—$1
Results tracked3931

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o1-mini leads

o1-mini: 35.5 (#224), Qwen2.5-Coder-32B: 22.6 (#333)

Benchmarko1-miniQwen2.5-Coder-32B
Aider Polyglot32.9%16.4%
LiveBench Coding48%56.9%
LMArena Coding13621276
HumanEval+89%87.2%
MBPP+78.8%77%
SWE-bench Verified (bash only)—9%
WeirdML36.3%—
BigCodeBench Instruct—49%
BigCodeBench Complete—58%

Agentic & Tool Use Not comparable

o1-mini: 24.6 (#118), Qwen2.5-Coder-32B: —

Benchmarko1-miniQwen2.5-Coder-32B
Cybench10%—

Reasoning Qwen2.5-Coder-32B leads

o1-mini: 8.8 (#346), Qwen2.5-Coder-32B: 21.2 (#225)

Benchmarko1-miniQwen2.5-Coder-32B
LiveBench Reasoning72.3%42.1%
LMArena Hard Prompts13331251
LiveBench Data Analysis57.9%49.9%
Epoch Capabilities Index135.82119.49
LiveBench57.8%46.2%
ARC-AGI-20.8%—
SimpleBench18.1%—
ARC-AGI-114%—
HellaSwag—83%
WinoGrande—80.8%

Math o1-mini leads

o1-mini: 35.4 (#186), Qwen2.5-Coder-32B: 33.3 (#204)

Benchmarko1-miniQwen2.5-Coder-32B
LiveBench Math62%46.6%
LMArena Math13581251
OTIS Mock AIME 2024-202546.9%—
MATH Level 589.2%—
FrontierMath (Feb 2025 set)1.7%—
GSM8K—93%

Knowledge o1-mini leads

o1-mini: 34.9 (#192), Qwen2.5-Coder-32B: 33.4 (#203)

Benchmarko1-miniQwen2.5-Coder-32B
LMArena Expert13161221
GPQA Diamond62.4%—
Confabulations18.6%—
ARC (AI2) Challenge—70.5%
MMLU—79.1%

Multilingual o1-mini leads

o1-mini: 43.6 (#182), Qwen2.5-Coder-32B: 37.8 (#235)

Benchmarko1-miniQwen2.5-Coder-32B
LMArena Non-English12891205
LMArena Chinese13141222
LMArena Russian12831228
LMArena French1293—
LMArena German1278—
LMArena Japanese1245—
LMArena Korean1223—
LMArena Spanish1303—

Instruction Following o1-mini leads

o1-mini: 66.7 (#206), Qwen2.5-Coder-32B: 61.4 (#245)

Benchmarko1-miniQwen2.5-Coder-32B
LiveBench Instruction Following65.4%58.7%
LMArena Instruction Following13041223

Long Context o1-mini leads

o1-mini: 40.1 (#161), Qwen2.5-Coder-32B: 38.0 (#208)

Benchmarko1-miniQwen2.5-Coder-32B
LMArena Longer Query13201251

Writing & Preference o1-mini leads

o1-mini: 48.4 (#202), Qwen2.5-Coder-32B: 41.6 (#240)

Benchmarko1-miniQwen2.5-Coder-32B
LMArena Text13171230
LMArena Creative Writing12441174
LMArena Multi-Turn13141222
LiveBench Language40.9%23.3%
Short-Story Creative Writing64.9%—

Frequently asked questions

Is o1-mini better than Qwen2.5-Coder-32B?

o1-mini and Qwen2.5-Coder-32B score almost the same on the Noometry Index (34.0 vs 33.4), so choose on price, context window or the category you care about most.

Is o1-mini or Qwen2.5-Coder-32B better for coding?

o1-mini scores higher on coding benchmarks: 35.5 versus 22.6 in the Noometry coding category.

How many benchmarks do o1-mini and Qwen2.5-Coder-32B share?

23 benchmarks have published results for both models. o1-mini has 39 scored results on Noometry and Qwen2.5-Coder-32B has 31.

Related comparisons

Go deeper