Noometry

Model comparison

Gemma 2 9B vs Granite 3.1 8b Instruct

Granite 3.1 8b Instruct is the stronger model overall, scoring 32.4 to 25.9 on the Noometry Index.

Last verified . 12 shared benchmarks.

Gemma 2 9B Google

25.9

Rank #341 Confirmed

Granite 3.1 8b Instruct IBM

32.4

Rank #258 Confirmed

Summary

  • They share 12 benchmarks with published results for both. Gemma 2 9B scores higher in 2 categories and Granite 3.1 8b Instruct in 6 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where Granite 3.1 8b Instruct leads 33.0 to 9.9.

Side by side

Gemma 2 9BGranite 3.1 8b Instruct
ProviderGoogleIBM
Noometry Index25.932.4
Released2024-06-24—
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked3513

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Granite 3.1 8b Instruct leads

Gemma 2 9B: 29.4 (#304), Granite 3.1 8b Instruct: 34.5 (#233)

BenchmarkGemma 2 9BGranite 3.1 8b Instruct
LMArena Coding11731186
BigCodeBench Instruct34.7%—
LiveBench Coding22.5%—
BigCodeBench Complete40.6%—

Agentic & Tool Use Not comparable

Gemma 2 9B: —, Granite 3.1 8b Instruct: 24.1 (#120)

BenchmarkGemma 2 9BGranite 3.1 8b Instruct
Berkeley Function Calling Leaderboard—27.1%

Reasoning Granite 3.1 8b Instruct leads

Gemma 2 9B: 15.9 (#309), Granite 3.1 8b Instruct: 22.1 (#207)

BenchmarkGemma 2 9BGranite 3.1 8b Instruct
LMArena Hard Prompts11711145
LiveBench Reasoning15.2%—
LiveBench Data Analysis36.4%—
Epoch Capabilities Index119.83—
LiveBench28.7%—
PIQA83.7%—

Math Granite 3.1 8b Instruct leads

Gemma 2 9B: 9.9 (#318), Granite 3.1 8b Instruct: 33.0 (#209)

BenchmarkGemma 2 9BGranite 3.1 8b Instruct
LMArena Math11831152
OTIS Mock AIME 2024-20250.6%—
LiveBench Math19.8%—
MATH Level 521%—
GSM8K84.9%—

Knowledge Granite 3.1 8b Instruct leads

Gemma 2 9B: 9.7 (#305), Granite 3.1 8b Instruct: 31.1 (#220)

BenchmarkGemma 2 9BGranite 3.1 8b Instruct
LMArena Expert11471142
GPQA Diamond27.5%—
BoolQ85.7%—
MMLU72.1%—

Multilingual Gemma 2 9B leads

Gemma 2 9B: 36.6 (#238), Granite 3.1 8b Instruct: 30.9 (#260)

BenchmarkGemma 2 9BGranite 3.1 8b Instruct
LMArena Non-English11881099
LMArena Chinese11851145
LMArena Russian12001092
LMArena French1190—
LMArena German1186—
LMArena Japanese1144—
LMArena Korean1137—
LMArena Spanish1200—

Instruction Following Granite 3.1 8b Instruct leads

Gemma 2 9B: 57.6 (#269), Granite 3.1 8b Instruct: 58.6 (#259)

BenchmarkGemma 2 9BGranite 3.1 8b Instruct
LMArena Instruction Following11781131
LiveBench Instruction Following52.6%—

Long Context Gemma 2 9B leads

Gemma 2 9B: 36.3 (#233), Granite 3.1 8b Instruct: 35.2 (#241)

BenchmarkGemma 2 9BGranite 3.1 8b Instruct
LMArena Longer Query11971162

Writing & Preference Granite 3.1 8b Instruct leads

Gemma 2 9B: 32.1 (#281), Granite 3.1 8b Instruct: 35.5 (#266)

BenchmarkGemma 2 9BGranite 3.1 8b Instruct
LMArena Text12071150
LMArena Creative Writing12061129
LMArena Multi-Turn11931108
EQ-Bench Creative Writing841—
LiveBench Language25.5%—

Frequently asked questions

Is Gemma 2 9B better than Granite 3.1 8b Instruct?

Granite 3.1 8b Instruct is the stronger model overall, scoring 32.4 to 25.9 on the Noometry Index.

Is Gemma 2 9B or Granite 3.1 8b Instruct better for coding?

Granite 3.1 8b Instruct scores higher on coding benchmarks: 34.5 versus 29.4 in the Noometry coding category.

How many benchmarks do Gemma 2 9B and Granite 3.1 8b Instruct share?

12 benchmarks have published results for both models. Gemma 2 9B has 35 scored results on Noometry and Granite 3.1 8b Instruct has 13.

Related comparisons

Go deeper