Noometry

Model comparison

Gemma 2 27B vs Grok-2 (Dec 2024)

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.4 to 29.2 on the Noometry Index.

Last verified . 30 shared benchmarks.

Gemma 2 27B Google

29.2

Rank #302 Confirmed

Grok-2 (Dec 2024) xAI

33.4

Rank #233 Confirmed

Summary

  • They share 30 benchmarks with published results for both. Gemma 2 27B scores higher in 1 category and Grok-2 (Dec 2024) in 7 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Grok-2 (Dec 2024) leads 29.8 to 18.9.
  • The biggest single-benchmark swing is MATH Level 5: 27.9% for Gemma 2 27B and 63.5% for Grok-2 (Dec 2024).
  • Gemma 2 27B has downloadable open weights; the other is API-only.

Side by side

Gemma 2 27BGrok-2 (Dec 2024)
ProviderGooglexAI
Noometry Index29.233.4
Released2024-06-242024-08-13
WeightsOpenProprietary
Context window8K—
Max output2K—
Input $ / M tokens$0.65—
Output $ / M tokens$0.65—
Results tracked3434

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Too close to call

Gemma 2 27B: 33.5 (#243), Grok-2 (Dec 2024): 32.9 (#254)

BenchmarkGemma 2 27BGrok-2 (Dec 2024)
LiveBench Coding36%46.4%
LMArena Coding12111288
WeirdML—22.2%
BigCodeBench Instruct42.8%—
BigCodeBench Complete52.5%—

Reasoning Grok-2 (Dec 2024) leads

Gemma 2 27B: 14.8 (#313), Grok-2 (Dec 2024): 16.4 (#294)

BenchmarkGemma 2 27BGrok-2 (Dec 2024)
LiveBench Reasoning28.1%54.8%
LMArena Hard Prompts11991272
DTBench48%65.2%
LiveBench Data Analysis47.9%54.5%
Epoch Capabilities Index122.06130.48
LiveBench38.2%54.3%
SimpleBench—22.7%
LMCA7.1%—

Math Grok-2 (Dec 2024) leads

Gemma 2 27B: 10.0 (#306), Grok-2 (Dec 2024): 19.6 (#278)

BenchmarkGemma 2 27BGrok-2 (Dec 2024)
OTIS Mock AIME 2024-20251.4%11.5%
LiveBench Math26.5%54.9%
LMArena Math12121283
MATH Level 527.9%63.5%
FrontierMath (Feb 2025 set)—0.7%

Knowledge Grok-2 (Dec 2024) leads

Gemma 2 27B: 18.9 (#275), Grok-2 (Dec 2024): 29.8 (#225)

BenchmarkGemma 2 27BGrok-2 (Dec 2024)
GPQA Diamond36.5%53.8%
Confabulations27.1%20.1%
LMArena Expert11721254
MMLU75.7%—

Multilingual Grok-2 (Dec 2024) leads

Gemma 2 27B: 38.8 (#221), Grok-2 (Dec 2024): 43.3 (#183)

BenchmarkGemma 2 27BGrok-2 (Dec 2024)
LMArena Non-English12171282
LMArena Chinese12211289
LMArena French12461317
LMArena German12091287
LMArena Japanese11751244
LMArena Korean11741237
LMArena Russian12341286
LMArena Spanish12281281

Instruction Following Grok-2 (Dec 2024) leads

Gemma 2 27B: 60.7 (#244), Grok-2 (Dec 2024): 67.1 (#197)

BenchmarkGemma 2 27BGrok-2 (Dec 2024)
LiveBench Instruction Following58.1%69.6%
LMArena Instruction Following12061270

Long Context Grok-2 (Dec 2024) leads

Gemma 2 27B: 37.3 (#213), Grok-2 (Dec 2024): 38.7 (#186)

BenchmarkGemma 2 27BGrok-2 (Dec 2024)
LMArena Longer Query12311276

Writing & Preference Grok-2 (Dec 2024) leads

Gemma 2 27B: 44.4 (#220), Grok-2 (Dec 2024): 48.8 (#192)

BenchmarkGemma 2 27BGrok-2 (Dec 2024)
LMArena Text12311305
LMArena Creative Writing12411284
LMArena Multi-Turn12241290
LiveBench Language32.6%45.6%
Short-Story Creative Writing—63.6%

Frequently asked questions

Is Gemma 2 27B better than Grok-2 (Dec 2024)?

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.4 to 29.2 on the Noometry Index.

Is Gemma 2 27B or Grok-2 (Dec 2024) better for coding?

They score almost the same on coding (33.5 vs 32.9); test both on your own repository before choosing.

How many benchmarks do Gemma 2 27B and Grok-2 (Dec 2024) share?

30 benchmarks have published results for both models. Gemma 2 27B has 34 scored results on Noometry and Grok-2 (Dec 2024) has 34.

Related comparisons

Go deeper