Noometry

Model comparison

Gemma 2 9B vs Grok-2 (Dec 2024)

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.7 to 25.9 on the Noometry Index.

Last verified . 28 shared benchmarks.

Gemma 2 9B Google

25.9

Rank #341 Confirmed

Grok-2 (Dec 2024) xAI

33.7

Rank #239 Confirmed

Summary

  • They share 28 benchmarks with published results for both. Gemma 2 9B scores higher in 0 categories and Grok-2 (Dec 2024) in 8 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Grok-2 (Dec 2024) leads 29.8 to 9.7.
  • The biggest single-benchmark swing is MATH Level 5: 21% for Gemma 2 9B and 63.5% for Grok-2 (Dec 2024).
  • Gemma 2 9B has downloadable open weights; the other is API-only.

Side by side

Gemma 2 9BGrok-2 (Dec 2024)
ProviderGooglexAI
Noometry Index25.933.7
Released2024-06-242024-08-13
WeightsOpenProprietary
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked3534

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Grok-2 (Dec 2024) leads

Gemma 2 9B: 29.4 (#304), Grok-2 (Dec 2024): 33.3 (#258)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
LiveBench Coding22.5%46.4%
LMArena Coding11731287
WeirdML—22.2%
BigCodeBench Instruct34.7%—
BigCodeBench Complete40.6%—

Reasoning Too close to call

Gemma 2 9B: 15.9 (#309), Grok-2 (Dec 2024): 16.9 (#299)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
LiveBench Reasoning15.2%54.8%
LMArena Hard Prompts11711272
LiveBench Data Analysis36.4%54.5%
Epoch Capabilities Index119.83130.48
LiveBench28.7%54.3%
SimpleBench—22.7%
DTBench—65.2%
PIQA83.7%—

Math Grok-2 (Dec 2024) leads

Gemma 2 9B: 9.9 (#318), Grok-2 (Dec 2024): 20.8 (#284)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
OTIS Mock AIME 2024-20250.6%11.5%
LiveBench Math19.8%54.9%
LMArena Math11831283
MATH Level 521%63.5%
FrontierMath (Feb 2025 set)—0.7%
GSM8K84.9%—

Knowledge Grok-2 (Dec 2024) leads

Gemma 2 9B: 9.7 (#305), Grok-2 (Dec 2024): 29.8 (#233)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
GPQA Diamond27.5%53.8%
LMArena Expert11471254
Confabulations—20.1%
BoolQ85.7%—
MMLU72.1%—

Multilingual Grok-2 (Dec 2024) leads

Gemma 2 9B: 36.6 (#238), Grok-2 (Dec 2024): 43.1 (#188)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
LMArena Non-English11881282
LMArena Chinese11851289
LMArena French11901318
LMArena German11861287
LMArena Japanese11441244
LMArena Korean11371237
LMArena Russian12001286
LMArena Spanish12001281

Instruction Following Grok-2 (Dec 2024) leads

Gemma 2 9B: 57.6 (#269), Grok-2 (Dec 2024): 66.9 (#202)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
LiveBench Instruction Following52.6%69.6%
LMArena Instruction Following11781270

Long Context Grok-2 (Dec 2024) leads

Gemma 2 9B: 36.3 (#233), Grok-2 (Dec 2024): 38.8 (#190)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
LMArena Longer Query11971276

Writing & Preference Grok-2 (Dec 2024) leads

Gemma 2 9B: 32.1 (#281), Grok-2 (Dec 2024): 48.6 (#198)

BenchmarkGemma 2 9BGrok-2 (Dec 2024)
LMArena Text12071305
LMArena Creative Writing12061284
LMArena Multi-Turn11931290
LiveBench Language25.5%45.6%
Short-Story Creative Writing—63.6%
EQ-Bench Creative Writing841—

Frequently asked questions

Is Gemma 2 9B better than Grok-2 (Dec 2024)?

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.7 to 25.9 on the Noometry Index.

Is Gemma 2 9B or Grok-2 (Dec 2024) better for coding?

Grok-2 (Dec 2024) scores higher on coding benchmarks: 33.3 versus 29.4 in the Noometry coding category.

How many benchmarks do Gemma 2 9B and Grok-2 (Dec 2024) share?

28 benchmarks have published results for both models. Gemma 2 9B has 35 scored results on Noometry and Grok-2 (Dec 2024) has 34.

Related comparisons

Go deeper