Noometry

Model comparison

Claude 3.5 Sonnet vs Qwen1.5-72B

Claude 3.5 Sonnet is the stronger model overall, scoring 34.4 to 30.3 on the Noometry Index.

Last verified . 22 shared benchmarks.

Claude 3.5 Sonnet Anthropic

34.4

Rank #225 Confirmed

Qwen1.5-72B Alibaba (Qwen)

30.3

Rank #282 Confirmed

Summary

  • They share 22 benchmarks with published results for both. Claude 3.5 Sonnet scores higher in 7 categories and Qwen1.5-72B in 1 category; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Claude 3.5 Sonnet leads 28.5 to 11.5.
  • The biggest single-benchmark swing is GPQA Diamond: 55.3% for Claude 3.5 Sonnet and 28.8% for Qwen1.5-72B.
  • Qwen1.5-72B has downloadable open weights; the other is API-only.

Side by side

Claude 3.5 SonnetQwen1.5-72B
ProviderAnthropicAlibaba (Qwen)
Noometry Index34.430.3
Released2024-06-202024-02-04
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked6022

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Claude 3.5 Sonnet leads

Claude 3.5 Sonnet: 39.5 (#145), Qwen1.5-72B: 31.4 (#272)

BenchmarkClaude 3.5 SonnetQwen1.5-72B
BigCodeBench Instruct46.8%33.2%
LMArena Coding13431166
BigCodeBench Complete58.6%40.3%
HumanEval+81.7%59.1%
MBPP+74.3%61.6%
Aider Polyglot51.6%—
GSO4.6%—
WeirdML40%—
LiveBench Coding67.1%—
CadEval48%—

Agentic & Tool Use Not comparable

Claude 3.5 Sonnet: 32.1 (#61), Qwen1.5-72B: —

BenchmarkClaude 3.5 SonnetQwen1.5-72B
TheAgentCompany24%—
Cybench17.5%—
BALROG32.6%—
METR Time Horizons45.2%—

Reasoning Claude 3.5 Sonnet leads

Claude 3.5 Sonnet: 22.5 (#176), Qwen1.5-72B: 21.2 (#201)

BenchmarkClaude 3.5 SonnetQwen1.5-72B
LMArena Hard Prompts13051148
SimpleBench41.4%—
EnigmaEval0.9%—
LiveBench Reasoning56.7%—
DTBench67.8%—
LiveBench Data Analysis55%—
Epoch Capabilities Index133.55—
ForecastBench60.7—
LiveBench59%—

Math Qwen1.5-72B leads

Claude 3.5 Sonnet: 18.0 (#282), Qwen1.5-72B: 30.7 (#205)

BenchmarkClaude 3.5 SonnetQwen1.5-72B
LMArena Math13061164
OTIS Mock AIME 2024-20258.5%—
Omni-MATH27.6%—
LiveBench Math52.3%—
MATH Level 556.9%—
FrontierMath (Feb 2025 set)2.1%—
FrontierMath Tier 4 (v1)0%—

Knowledge Claude 3.5 Sonnet leads

Claude 3.5 Sonnet: 28.5 (#236), Qwen1.5-72B: 11.5 (#295)

BenchmarkClaude 3.5 SonnetQwen1.5-72B
GPQA Diamond55.3%28.8%
LMArena Expert12641136
Humanity's Last Exam4.1%—
MMLU-Pro77.7%—
Confabulations19.9%—
GPQA (HELM)56.5%—
MMLU87.3%—

Multimodal Not comparable

Claude 3.5 Sonnet: 26.0 (#115), Qwen1.5-72B: —

BenchmarkClaude 3.5 SonnetQwen1.5-72B
LMArena Vision1125—
Video-MME60%—
GeoBench62%—
VPCT33%—

Multilingual Claude 3.5 Sonnet leads

Claude 3.5 Sonnet: 43.4 (#180), Qwen1.5-72B: 33.4 (#248)

BenchmarkClaude 3.5 SonnetQwen1.5-72B
LMArena Non-English12831135
LMArena Chinese12711186
LMArena French13041158
LMArena German12971084
LMArena Japanese12341061
LMArena Korean12001049
LMArena Russian13051104
LMArena Spanish12891110

Instruction Following Claude 3.5 Sonnet leads

Claude 3.5 Sonnet: 69.0 (#177), Qwen1.5-72B: 59.6 (#251)

BenchmarkClaude 3.5 SonnetQwen1.5-72B
LMArena Instruction Following12971142
LiveBench Instruction Following69.3%—
IFEval85.5%—

Long Context Claude 3.5 Sonnet leads

Claude 3.5 Sonnet: 39.9 (#162), Qwen1.5-72B: 35.0 (#238)

BenchmarkClaude 3.5 SonnetQwen1.5-72B
LMArena Longer Query13111157

Writing & Preference Claude 3.5 Sonnet leads

Claude 3.5 Sonnet: 53.0 (#159), Qwen1.5-72B: 37.4 (#253)

BenchmarkClaude 3.5 SonnetQwen1.5-72B
LMArena Text12981166
LMArena Creative Writing12921137
LMArena Multi-Turn13261160
Short-Story Creative Writing80.3%—
EQ-Bench Creative Writing1451—
WildBench79.2%—
LiveBench Language53.8%—

Frequently asked questions

Is Claude 3.5 Sonnet better than Qwen1.5-72B?

Claude 3.5 Sonnet is the stronger model overall, scoring 34.4 to 30.3 on the Noometry Index.

Is Claude 3.5 Sonnet or Qwen1.5-72B better for coding?

Claude 3.5 Sonnet scores higher on coding benchmarks: 39.5 versus 31.4 in the Noometry coding category.

How many benchmarks do Claude 3.5 Sonnet and Qwen1.5-72B share?

22 benchmarks have published results for both models. Claude 3.5 Sonnet has 60 scored results on Noometry and Qwen1.5-72B has 22.

Related comparisons

Go deeper