Noometry

Model comparison

Granite 3.1 8b Instruct vs Grok-2 (Dec 2024)

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.7 to 32.4 on the Noometry Index.

Last verified . 12 shared benchmarks.

Granite 3.1 8b Instruct IBM

32.4

Rank #258 Confirmed

Grok-2 (Dec 2024) xAI

33.7

Rank #239 Confirmed

Summary

  • They share 12 benchmarks with published results for both. Granite 3.1 8b Instruct scores higher in 4 categories and Grok-2 (Dec 2024) in 4 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Grok-2 (Dec 2024) leads 48.6 to 35.5.
  • Granite 3.1 8b Instruct has downloadable open weights; the other is API-only.

Side by side

Granite 3.1 8b InstructGrok-2 (Dec 2024)
ProviderIBMxAI
Noometry Index32.433.7
Released—2024-08-13
WeightsOpenProprietary
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked1334

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 34.5 (#233), Grok-2 (Dec 2024): 33.3 (#258)

BenchmarkGranite 3.1 8b InstructGrok-2 (Dec 2024)
LMArena Coding11861287
WeirdML—22.2%
LiveBench Coding—46.4%

Agentic & Tool Use Not comparable

Granite 3.1 8b Instruct: 24.1 (#120), Grok-2 (Dec 2024): —

BenchmarkGranite 3.1 8b InstructGrok-2 (Dec 2024)
Berkeley Function Calling Leaderboard27.1%—

Reasoning Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 22.1 (#207), Grok-2 (Dec 2024): 16.9 (#299)

BenchmarkGranite 3.1 8b InstructGrok-2 (Dec 2024)
LMArena Hard Prompts11451272
SimpleBench—22.7%
LiveBench Reasoning—54.8%
DTBench—65.2%
LiveBench Data Analysis—54.5%
Epoch Capabilities Index—130.48
LiveBench—54.3%

Math Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 33.0 (#209), Grok-2 (Dec 2024): 20.8 (#284)

BenchmarkGranite 3.1 8b InstructGrok-2 (Dec 2024)
LMArena Math11521283
OTIS Mock AIME 2024-2025—11.5%
LiveBench Math—54.9%
MATH Level 5—63.5%
FrontierMath (Feb 2025 set)—0.7%

Knowledge Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 31.1 (#220), Grok-2 (Dec 2024): 29.8 (#233)

BenchmarkGranite 3.1 8b InstructGrok-2 (Dec 2024)
LMArena Expert11421254
GPQA Diamond—53.8%
Confabulations—20.1%

Multilingual Grok-2 (Dec 2024) leads

Granite 3.1 8b Instruct: 30.9 (#260), Grok-2 (Dec 2024): 43.1 (#188)

BenchmarkGranite 3.1 8b InstructGrok-2 (Dec 2024)
LMArena Non-English10991282
LMArena Chinese11451289
LMArena Russian10921286
LMArena French—1318
LMArena German—1287
LMArena Japanese—1244
LMArena Korean—1237
LMArena Spanish—1281

Instruction Following Grok-2 (Dec 2024) leads

Granite 3.1 8b Instruct: 58.6 (#259), Grok-2 (Dec 2024): 66.9 (#202)

BenchmarkGranite 3.1 8b InstructGrok-2 (Dec 2024)
LMArena Instruction Following11311270
LiveBench Instruction Following—69.6%

Long Context Grok-2 (Dec 2024) leads

Granite 3.1 8b Instruct: 35.2 (#241), Grok-2 (Dec 2024): 38.8 (#190)

BenchmarkGranite 3.1 8b InstructGrok-2 (Dec 2024)
LMArena Longer Query11621276

Writing & Preference Grok-2 (Dec 2024) leads

Granite 3.1 8b Instruct: 35.5 (#266), Grok-2 (Dec 2024): 48.6 (#198)

BenchmarkGranite 3.1 8b InstructGrok-2 (Dec 2024)
LMArena Text11501305
LMArena Creative Writing11291284
LMArena Multi-Turn11081290
Short-Story Creative Writing—63.6%
LiveBench Language—45.6%

Frequently asked questions

Is Granite 3.1 8b Instruct better than Grok-2 (Dec 2024)?

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.7 to 32.4 on the Noometry Index.

Is Granite 3.1 8b Instruct or Grok-2 (Dec 2024) better for coding?

Granite 3.1 8b Instruct scores higher on coding benchmarks: 34.5 versus 33.3 in the Noometry coding category.

How many benchmarks do Granite 3.1 8b Instruct and Grok-2 (Dec 2024) share?

12 benchmarks have published results for both models. Granite 3.1 8b Instruct has 13 scored results on Noometry and Grok-2 (Dec 2024) has 34.

Related comparisons

Go deeper