Noometry

Model comparison

Claude 3.5 Haiku vs Qwen2.5-Coder-32B

Qwen2.5-Coder-32B is the stronger model overall, scoring 32.9 to 28.9 on the Noometry Index.

Last verified . 24 shared benchmarks.

Claude 3.5 Haiku Anthropic

28.9

Rank #309 Confirmed

Qwen2.5-Coder-32B Alibaba (Qwen)

32.9

Rank #239 Confirmed

Summary

  • They share 24 benchmarks with published results for both. Claude 3.5 Haiku scores higher in 5 categories and Qwen2.5-Coder-32B in 3 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in math, where Qwen2.5-Coder-32B leads 30.8 to 13.7.
  • The biggest single-benchmark swing is LiveBench Reasoning: 28.1% for Claude 3.5 Haiku and 42.1% for Qwen2.5-Coder-32B.
  • Qwen2.5-Coder-32B has downloadable open weights; the other is API-only.

Side by side

Claude 3.5 HaikuQwen2.5-Coder-32B
ProviderAnthropicAlibaba (Qwen)
Noometry Index28.932.9
Released2024-10-222024-09-18
WeightsProprietaryOpen
Context window—33K
Max output—29K
Input $ / M tokens—$0.66
Output $ / M tokens—$1
Results tracked4931

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Claude 3.5 Haiku leads

Claude 3.5 Haiku: 32.6 (#258), Qwen2.5-Coder-32B: 22.4 (#328)

BenchmarkClaude 3.5 HaikuQwen2.5-Coder-32B
Aider Polyglot28%16.4%
BigCodeBench Instruct46.1%49%
LiveBench Coding51.4%56.9%
LMArena Coding12871276
BigCodeBench Complete59%58%
SWE-bench Verified (bash only)—9%
SciCode27.4%—
WeirdML30.7%—
CadEval32%—
HumanEval+—87.2%
MBPP+—77%

Agentic & Tool Use Not comparable

Claude 3.5 Haiku: 27.7 (#90), Qwen2.5-Coder-32B: —

BenchmarkClaude 3.5 HaikuQwen2.5-Coder-32B
BALROG19.3%—

Reasoning Qwen2.5-Coder-32B leads

Claude 3.5 Haiku: 17.1 (#286), Qwen2.5-Coder-32B: 20.4 (#220)

BenchmarkClaude 3.5 HaikuQwen2.5-Coder-32B
LiveBench Reasoning28.1%42.1%
LMArena Hard Prompts12511251
LiveBench Data Analysis48.5%49.9%
Epoch Capabilities Index127.15119.41
LiveBench43.5%46.2%
CritPt0%—
DTBench56.7%—
HellaSwag—83%
WinoGrande—80.8%

Math Qwen2.5-Coder-32B leads

Claude 3.5 Haiku: 13.7 (#294), Qwen2.5-Coder-32B: 30.8 (#203)

BenchmarkClaude 3.5 HaikuQwen2.5-Coder-32B
LiveBench Math35.5%46.6%
LMArena Math12441251
OTIS Mock AIME 2024-20254.3%—
Omni-MATH22.4%—
MATH Level 546.4%—
FrontierMath (Feb 2025 set)0.3%—
GSM8K—93%

Knowledge Qwen2.5-Coder-32B leads

Claude 3.5 Haiku: 18.7 (#276), Qwen2.5-Coder-32B: 32.9 (#200)

BenchmarkClaude 3.5 HaikuQwen2.5-Coder-32B
LMArena Expert12081221
MMLU74.3%79.1%
GPQA Diamond38.1%—
MMLU-Pro60.5%—
Confabulations36.7%—
GPQA (HELM)36.3%—
ARC (AI2) Challenge—70.5%

Multimodal Not comparable

Claude 3.5 Haiku: 26.3 (#113), Qwen2.5-Coder-32B: —

BenchmarkClaude 3.5 HaikuQwen2.5-Coder-32B
LMArena Vision1092—
GeoBench34%—

Multilingual Claude 3.5 Haiku leads

Claude 3.5 Haiku: 40.2 (#213), Qwen2.5-Coder-32B: 38.0 (#230)

BenchmarkClaude 3.5 HaikuQwen2.5-Coder-32B
LMArena Non-English12381205
LMArena Chinese12291222
LMArena Russian12531228
LMArena French1262—
LMArena German1237—
LMArena Japanese1175—
LMArena Korean1172—
LMArena Spanish1261—

Instruction Following Claude 3.5 Haiku leads

Claude 3.5 Haiku: 63.1 (#229), Qwen2.5-Coder-32B: 61.6 (#240)

BenchmarkClaude 3.5 HaikuQwen2.5-Coder-32B
LiveBench Instruction Following61.9%58.7%
LMArena Instruction Following12411223
IFEval79.2%—

Long Context Too close to call

Claude 3.5 Haiku: 38.3 (#195), Qwen2.5-Coder-32B: 37.9 (#204)

BenchmarkClaude 3.5 HaikuQwen2.5-Coder-32B
LMArena Longer Query12611251

Writing & Preference Claude 3.5 Haiku leads

Claude 3.5 Haiku: 42.9 (#228), Qwen2.5-Coder-32B: 41.8 (#235)

BenchmarkClaude 3.5 HaikuQwen2.5-Coder-32B
LMArena Text12551230
LMArena Creative Writing12331174
LMArena Multi-Turn12651222
LiveBench Language35.4%23.3%
Short-Story Creative Writing73.5%—
EQ-Bench Creative Writing1146—
WildBench76%—

Frequently asked questions

Is Claude 3.5 Haiku better than Qwen2.5-Coder-32B?

Qwen2.5-Coder-32B is the stronger model overall, scoring 32.9 to 28.9 on the Noometry Index.

Is Claude 3.5 Haiku or Qwen2.5-Coder-32B better for coding?

Claude 3.5 Haiku scores higher on coding benchmarks: 32.6 versus 22.4 in the Noometry coding category.

How many benchmarks do Claude 3.5 Haiku and Qwen2.5-Coder-32B share?

24 benchmarks have published results for both models. Claude 3.5 Haiku has 49 scored results on Noometry and Qwen2.5-Coder-32B has 31.

Related comparisons

Go deeper