Noometry

Model comparison

Claude 3 Opus vs Qwen1.5 4b Chat

Claude 3 Opus and Qwen1.5 4b Chat score almost the same on the Noometry Index (29.5 vs 28.8), so choose on price, context window or the category you care about most.

Last verified . 13 shared benchmarks.

Claude 3 Opus Anthropic

29.5

Rank #310 Confirmed

Qwen1.5 4b Chat Alibaba (Qwen)

28.8

Rank #322 Confirmed

Summary

  • They share 13 benchmarks with published results for both. Claude 3 Opus scores higher in 5 categories and Qwen1.5 4b Chat in 3 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Claude 3 Opus leads 47.2 to 23.8.
  • Qwen1.5 4b Chat has downloadable open weights; the other is API-only.

Side by side

Claude 3 OpusQwen1.5 4b Chat
ProviderAnthropicAlibaba (Qwen)
Noometry Index29.528.8
Released2024-02-29—
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked4613

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Claude 3 Opus leads

Claude 3 Opus: 32.9 (#267), Qwen1.5 4b Chat: 29.1 (#308)

BenchmarkClaude 3 OpusQwen1.5 4b Chat
LMArena Coding1264999
WeirdML19.2%—
BigCodeBench Instruct45.5%—
LiveBench Coding38.6%—
BigCodeBench Complete57.4%—
HumanEval+77.4%—
MBPP+73.3%—

Agentic & Tool Use Not comparable

Claude 3 Opus: 24.6 (#116), Qwen1.5 4b Chat: —

BenchmarkClaude 3 OpusQwen1.5 4b Chat
Cybench10%—
METR Time Horizons29.5%—

Reasoning Qwen1.5 4b Chat leads

Claude 3 Opus: 14.6 (#324), Qwen1.5 4b Chat: 18.5 (#279)

BenchmarkClaude 3 OpusQwen1.5 4b Chat
LMArena Hard Prompts1245976
SimpleBench23.5%—
Chess Puzzles5%—
EnigmaEval0.8%—
LiveBench Reasoning40.6%—
DTBench61.6%—
LiveBench Data Analysis57.9%—
LMCA17%—
Epoch Capabilities Index126.91—
ForecastBench58.4—
LiveBench49.2%—
WinoGrande88.5%—

Math Qwen1.5 4b Chat leads

Claude 3 Opus: 14.8 (#299), Qwen1.5 4b Chat: 30.4 (#234)

BenchmarkClaude 3 OpusQwen1.5 4b Chat
LMArena Math12731026
OTIS Mock AIME 2024-20254.7%—
LiveBench Math43.6%—
MATH Level 537.5%—

Knowledge Qwen1.5 4b Chat leads

Claude 3 Opus: 24.5 (#267), Qwen1.5 4b Chat: 26.7 (#255)

BenchmarkClaude 3 OpusQwen1.5 4b Chat
LMArena Expert1223980
GPQA Diamond47.2%—
SimpleQA Verified12.6%—
Confabulations22.7%—
MMLU84.6%—

Multimodal Not comparable

Claude 3 Opus: 27.1 (#116), Qwen1.5 4b Chat: —

BenchmarkClaude 3 OpusQwen1.5 4b Chat
LMArena Vision1023—

Multilingual Claude 3 Opus leads

Claude 3 Opus: 41.4 (#207), Qwen1.5 4b Chat: 24.1 (#290)

BenchmarkClaude 3 OpusQwen1.5 4b Chat
LMArena Non-English1258979
LMArena Chinese12481024
LMArena German1258902
LMArena Russian1280952
LMArena French1275—
LMArena Japanese1204—
LMArena Korean1187—
LMArena Spanish1246—

Instruction Following Claude 3 Opus leads

Claude 3 Opus: 64.1 (#228), Qwen1.5 4b Chat: 49.0 (#300)

BenchmarkClaude 3 OpusQwen1.5 4b Chat
LMArena Instruction Following1248978
LiveBench Instruction Following63.9%—

Long Context Claude 3 Opus leads

Claude 3 Opus: 38.2 (#202), Qwen1.5 4b Chat: 30.1 (#290)

BenchmarkClaude 3 OpusQwen1.5 4b Chat
LMArena Longer Query1259988

Writing & Preference Claude 3 Opus leads

Claude 3 Opus: 47.2 (#213), Qwen1.5 4b Chat: 23.8 (#309)

BenchmarkClaude 3 OpusQwen1.5 4b Chat
LMArena Text1262997
LMArena Creative Writing1235969
LMArena Multi-Turn1275977
LiveBench Language50.4%—

Frequently asked questions

Is Claude 3 Opus better than Qwen1.5 4b Chat?

Claude 3 Opus and Qwen1.5 4b Chat score almost the same on the Noometry Index (29.5 vs 28.8), so choose on price, context window or the category you care about most.

Is Claude 3 Opus or Qwen1.5 4b Chat better for coding?

Claude 3 Opus scores higher on coding benchmarks: 32.9 versus 29.1 in the Noometry coding category.

How many benchmarks do Claude 3 Opus and Qwen1.5 4b Chat share?

13 benchmarks have published results for both models. Claude 3 Opus has 46 scored results on Noometry and Qwen1.5 4b Chat has 13.

Related comparisons

Go deeper