Noometry

Model comparison

Granite 3.0 8b Instruct vs Qwen2.5-Coder-32B

Qwen2.5-Coder-32B is the stronger model overall, scoring 33.4 to 31.6 on the Noometry Index.

Last verified . 14 shared benchmarks.

Granite 3.0 8b Instruct IBM

31.6

Rank #270 Confirmed

Qwen2.5-Coder-32B Alibaba (Qwen)

33.4

Rank #245 Confirmed

Summary

  • They share 14 benchmarks with published results for both. Granite 3.0 8b Instruct scores higher in 1 category and Qwen2.5-Coder-32B in 7 categories; 6 gaps are clear of the uncertainty.
  • The widest gap is in multilingual, where Qwen2.5-Coder-32B leads 37.8 to 27.2.
  • The biggest single-benchmark swing is BigCodeBench Complete: 35.4% for Granite 3.0 8b Instruct and 58% for Qwen2.5-Coder-32B.

Side by side

Granite 3.0 8b InstructQwen2.5-Coder-32B
ProviderIBMAlibaba (Qwen)
Noometry Index31.633.4
Released—2024-09-18
WeightsOpenOpen
Context window—33K
Max output—29K
Input $ / M tokens—$0.66
Output $ / M tokens—$1
Results tracked1431

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Granite 3.0 8b Instruct leads

Granite 3.0 8b Instruct: 29.7 (#301), Qwen2.5-Coder-32B: 22.6 (#333)

BenchmarkGranite 3.0 8b InstructQwen2.5-Coder-32B
BigCodeBench Instruct29.3%49%
LMArena Coding11121276
BigCodeBench Complete35.4%58%
SWE-bench Verified (bash only)—9%
Aider Polyglot—16.4%
LiveBench Coding—56.9%
HumanEval+—87.2%
MBPP+—77%

Reasoning Too close to call

Granite 3.0 8b Instruct: 20.9 (#230), Qwen2.5-Coder-32B: 21.2 (#225)

BenchmarkGranite 3.0 8b InstructQwen2.5-Coder-32B
LMArena Hard Prompts10921251
LiveBench Reasoning—42.1%
LiveBench Data Analysis—49.9%
Epoch Capabilities Index—119.49
HellaSwag—83%
LiveBench—46.2%
WinoGrande—80.8%

Math Too close to call

Granite 3.0 8b Instruct: 32.8 (#210), Qwen2.5-Coder-32B: 33.3 (#204)

BenchmarkGranite 3.0 8b InstructQwen2.5-Coder-32B
LMArena Math11431251
LiveBench Math—46.6%
GSM8K—93%

Knowledge Qwen2.5-Coder-32B leads

Granite 3.0 8b Instruct: 29.6 (#236), Qwen2.5-Coder-32B: 33.4 (#203)

BenchmarkGranite 3.0 8b InstructQwen2.5-Coder-32B
LMArena Expert10871221
ARC (AI2) Challenge—70.5%
MMLU—79.1%

Multilingual Qwen2.5-Coder-32B leads

Granite 3.0 8b Instruct: 27.2 (#276), Qwen2.5-Coder-32B: 37.8 (#235)

BenchmarkGranite 3.0 8b InstructQwen2.5-Coder-32B
LMArena Non-English10371205
LMArena Chinese10631222
LMArena Russian10601228

Instruction Following Qwen2.5-Coder-32B leads

Granite 3.0 8b Instruct: 56.0 (#276), Qwen2.5-Coder-32B: 61.4 (#245)

BenchmarkGranite 3.0 8b InstructQwen2.5-Coder-32B
LMArena Instruction Following10881223
LiveBench Instruction Following—58.7%

Long Context Qwen2.5-Coder-32B leads

Granite 3.0 8b Instruct: 34.0 (#252), Qwen2.5-Coder-32B: 38.0 (#208)

BenchmarkGranite 3.0 8b InstructQwen2.5-Coder-32B
LMArena Longer Query11221251

Writing & Preference Qwen2.5-Coder-32B leads

Granite 3.0 8b Instruct: 31.1 (#285), Qwen2.5-Coder-32B: 41.6 (#240)

BenchmarkGranite 3.0 8b InstructQwen2.5-Coder-32B
LMArena Text10961230
LMArena Creative Writing10711174
LMArena Multi-Turn10631222
LiveBench Language—23.3%

Frequently asked questions

Is Granite 3.0 8b Instruct better than Qwen2.5-Coder-32B?

Qwen2.5-Coder-32B is the stronger model overall, scoring 33.4 to 31.6 on the Noometry Index.

Is Granite 3.0 8b Instruct or Qwen2.5-Coder-32B better for coding?

Granite 3.0 8b Instruct scores higher on coding benchmarks: 29.7 versus 22.6 in the Noometry coding category.

How many benchmarks do Granite 3.0 8b Instruct and Qwen2.5-Coder-32B share?

14 benchmarks have published results for both models. Granite 3.0 8b Instruct has 14 scored results on Noometry and Qwen2.5-Coder-32B has 31.

Related comparisons

Go deeper