Noometry

Model comparison

Claude 3.5 Haiku vs Qwen2-72B

Claude 3.5 Haiku and Qwen2-72B score almost the same on the Noometry Index (29.2 vs 30.0), so choose on price, context window or the category you care about most.

Last verified . 24 shared benchmarks.

Claude 3.5 Haiku Anthropic

29.2

Rank #315 Confirmed

Qwen2-72B Alibaba (Qwen)

30.0

Rank #300 Confirmed

Summary

  • They share 24 benchmarks with published results for both. Claude 3.5 Haiku scores higher in 6 categories and Qwen2-72B in 3 categories; 9 gaps are clear of the uncertainty.
  • The widest gap is in math, where Qwen2-72B leads 30.2 to 14.7.
  • The biggest single-benchmark swing is WeirdML: 30.7% for Claude 3.5 Haiku and 11.3% for Qwen2-72B.
  • Qwen2-72B has downloadable open weights; the other is API-only.

Side by side

Claude 3.5 HaikuQwen2-72B
ProviderAnthropicAlibaba (Qwen)
Noometry Index29.230.0
Released2024-10-222024-06-07
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked4926

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Claude 3.5 Haiku leads

Claude 3.5 Haiku: 32.9 (#265), Qwen2-72B: 29.1 (#310)

BenchmarkClaude 3.5 HaikuQwen2-72B
WeirdML30.7%11.3%
BigCodeBench Instruct46.1%38.5%
LMArena Coding12861196
BigCodeBench Complete59%54%
Aider Polyglot28%—
SciCode27.4%—
LiveBench Coding51.4%—
CadEval32%—

Agentic & Tool Use Claude 3.5 Haiku leads

Claude 3.5 Haiku: 28.0 (#95), Qwen2-72B: 17.0 (#146)

BenchmarkClaude 3.5 HaikuQwen2-72B
TheAgentCompany—1.1%
BALROG19.3%—
METR Time Horizons—29.9%

Reasoning Qwen2-72B leads

Claude 3.5 Haiku: 17.7 (#290), Qwen2-72B: 23.2 (#181)

BenchmarkClaude 3.5 HaikuQwen2-72B
LMArena Hard Prompts12511191
Epoch Capabilities Index127.15125.28
CritPt0%—
LiveBench Reasoning28.1%—
DTBench56.7%—
LiveBench Data Analysis48.5%—
LiveBench43.5%—

Math Qwen2-72B leads

Claude 3.5 Haiku: 14.7 (#300), Qwen2-72B: 30.2 (#236)

BenchmarkClaude 3.5 HaikuQwen2-72B
LMArena Math12441235
MATH Level 546.4%39.1%
OTIS Mock AIME 2024-20254.3%—
Omni-MATH22.4%—
LiveBench Math35.5%—
FrontierMath (Feb 2025 set)0.3%—

Knowledge Qwen2-72B leads

Claude 3.5 Haiku: 18.7 (#281), Qwen2-72B: 21.2 (#275)

BenchmarkClaude 3.5 HaikuQwen2-72B
GPQA Diamond38.1%40.8%
LMArena Expert12081171
MMLU74.3%82.4%
MMLU-Pro60.5%—
Confabulations36.7%—
GPQA (HELM)36.3%—

Multimodal Not comparable

Claude 3.5 Haiku: 26.8 (#117), Qwen2-72B: —

BenchmarkClaude 3.5 HaikuQwen2-72B
LMArena Vision1092—
GeoBench34%—

Multilingual Claude 3.5 Haiku leads

Claude 3.5 Haiku: 40.0 (#218), Qwen2-72B: 35.9 (#244)

BenchmarkClaude 3.5 HaikuQwen2-72B
LMArena Non-English12381176
LMArena Chinese12291240
LMArena French12641170
LMArena German12371151
LMArena Japanese11751111
LMArena Korean11731083
LMArena Russian12531169
LMArena Spanish12611169

Instruction Following Claude 3.5 Haiku leads

Claude 3.5 Haiku: 62.9 (#234), Qwen2-72B: 61.7 (#241)

BenchmarkClaude 3.5 HaikuQwen2-72B
LMArena Instruction Following12411181
LiveBench Instruction Following61.9%—
IFEval79.2%—

Long Context Claude 3.5 Haiku leads

Claude 3.5 Haiku: 38.3 (#200), Qwen2-72B: 36.1 (#235)

BenchmarkClaude 3.5 HaikuQwen2-72B
LMArena Longer Query12611192

Writing & Preference Claude 3.5 Haiku leads

Claude 3.5 Haiku: 42.7 (#234), Qwen2-72B: 40.8 (#241)

BenchmarkClaude 3.5 HaikuQwen2-72B
LMArena Text12551203
LMArena Creative Writing12331181
LMArena Multi-Turn12651196
Short-Story Creative Writing73.5%—
EQ-Bench Creative Writing1146—
WildBench76%—
LiveBench Language35.4%—

Frequently asked questions

Is Claude 3.5 Haiku better than Qwen2-72B?

Claude 3.5 Haiku and Qwen2-72B score almost the same on the Noometry Index (29.2 vs 30.0), so choose on price, context window or the category you care about most.

Is Claude 3.5 Haiku or Qwen2-72B better for coding?

Claude 3.5 Haiku scores higher on coding benchmarks: 32.9 versus 29.1 in the Noometry coding category.

How many benchmarks do Claude 3.5 Haiku and Qwen2-72B share?

24 benchmarks have published results for both models. Claude 3.5 Haiku has 49 scored results on Noometry and Qwen2-72B has 26.

Related comparisons

Go deeper