Noometry

Model comparison

Llama 3-8B vs Qwen2.5-Coder-32B

Qwen2.5-Coder-32B is the stronger model overall, scoring 33.4 to 25.5 on the Noometry Index.

Last verified . 20 shared benchmarks.

Llama 3-8B Meta

25.5

Rank #344 Confirmed

Qwen2.5-Coder-32B Alibaba (Qwen)

33.4

Rank #245 Confirmed

Summary

  • They share 20 benchmarks with published results for both. Llama 3-8B scores higher in 1 category and Qwen2.5-Coder-32B in 7 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Qwen2.5-Coder-32B leads 33.4 to 7.8.
  • The biggest single-benchmark swing is BigCodeBench Complete: 36.9% for Llama 3-8B and 58% for Qwen2.5-Coder-32B.

Side by side

Llama 3-8BQwen2.5-Coder-32B
ProviderMetaAlibaba (Qwen)
Noometry Index25.533.4
Released2024-04-182024-09-18
WeightsOpenOpen
Context window—33K
Max output—29K
Input $ / M tokens—$0.66
Output $ / M tokens—$1
Results tracked3431

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Llama 3-8B leads

Llama 3-8B: 31.0 (#289), Qwen2.5-Coder-32B: 22.6 (#333)

BenchmarkLlama 3-8BQwen2.5-Coder-32B
BigCodeBench Instruct31.9%49%
LMArena Coding11521276
BigCodeBench Complete36.9%58%
HumanEval+56.7%87.2%
MBPP+54.8%77%
SWE-bench Verified (bash only)—9%
Aider Polyglot—16.4%
LiveBench Coding—56.9%

Reasoning Qwen2.5-Coder-32B leads

Llama 3-8B: 14.3 (#326), Qwen2.5-Coder-32B: 21.2 (#225)

BenchmarkLlama 3-8BQwen2.5-Coder-32B
LMArena Hard Prompts11331251
Epoch Capabilities Index116.45119.49
WinoGrande75.7%80.8%
Chess Puzzles0%—
LiveBench Reasoning—42.1%
DTBench43.9%—
LiveBench Data Analysis—49.9%
Adversarial NLI57.3%—
ForecastBench58.6—
HellaSwag—83%
LiveBench—46.2%

Math Qwen2.5-Coder-32B leads

Llama 3-8B: 8.8 (#323), Qwen2.5-Coder-32B: 33.3 (#204)

BenchmarkLlama 3-8BQwen2.5-Coder-32B
LMArena Math11511251
OTIS Mock AIME 2024-20251.9%—
LiveBench Math—46.6%
MATH Level 56.1%—
GSM8K—93%

Knowledge Qwen2.5-Coder-32B leads

Llama 3-8B: 7.8 (#308), Qwen2.5-Coder-32B: 33.4 (#203)

BenchmarkLlama 3-8BQwen2.5-Coder-32B
LMArena Expert11131221
ARC (AI2) Challenge82.8%70.5%
MMLU68.8%79.1%
GPQA Diamond26.1%—
OpenBookQA82.6%—
TriviaQA67.7%—

Multilingual Qwen2.5-Coder-32B leads

Llama 3-8B: 30.8 (#261), Qwen2.5-Coder-32B: 37.8 (#235)

BenchmarkLlama 3-8BQwen2.5-Coder-32B
LMArena Non-English10981205
LMArena Chinese10761222
LMArena Russian11091228
LMArena French1159—
LMArena German1104—
LMArena Japanese967—
LMArena Korean1004—
LMArena Spanish1173—

Instruction Following Qwen2.5-Coder-32B leads

Llama 3-8B: 58.4 (#260), Qwen2.5-Coder-32B: 61.4 (#245)

BenchmarkLlama 3-8BQwen2.5-Coder-32B
LMArena Instruction Following11271223
LiveBench Instruction Following—58.7%

Long Context Qwen2.5-Coder-32B leads

Llama 3-8B: 34.2 (#251), Qwen2.5-Coder-32B: 38.0 (#208)

BenchmarkLlama 3-8BQwen2.5-Coder-32B
LMArena Longer Query11281251

Writing & Preference Qwen2.5-Coder-32B leads

Llama 3-8B: 37.5 (#256), Qwen2.5-Coder-32B: 41.6 (#240)

BenchmarkLlama 3-8BQwen2.5-Coder-32B
LMArena Text11661230
LMArena Creative Writing11501174
LMArena Multi-Turn11521222
LiveBench Language—23.3%

Frequently asked questions

Is Llama 3-8B better than Qwen2.5-Coder-32B?

Qwen2.5-Coder-32B is the stronger model overall, scoring 33.4 to 25.5 on the Noometry Index.

Is Llama 3-8B or Qwen2.5-Coder-32B better for coding?

Llama 3-8B scores higher on coding benchmarks: 31.0 versus 22.6 in the Noometry coding category.

How many benchmarks do Llama 3-8B and Qwen2.5-Coder-32B share?

20 benchmarks have published results for both models. Llama 3-8B has 34 scored results on Noometry and Qwen2.5-Coder-32B has 31.

Related comparisons

Go deeper