Noometry

Model comparison

Gemma 2 27B vs Llama 3-70B

Gemma 2 27B and Llama 3-70B score almost the same on the Noometry Index (29.4 vs 28.8), so choose on price, context window or the category you care about most.

Last verified . 25 shared benchmarks.

Gemma 2 27B Google

29.4

Rank #312 Confirmed

Llama 3-70B Meta

28.8

Rank #323 Confirmed

Summary

  • They share 25 benchmarks with published results for both. Gemma 2 27B scores higher in 3 categories and Llama 3-70B in 5 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in multilingual, where Gemma 2 27B leads 38.6 to 33.6.
  • The biggest single-benchmark swing is DTBench: 48% for Gemma 2 27B and 54.2% for Llama 3-70B.

Side by side

Gemma 2 27BLlama 3-70B
ProviderGoogleMeta
Noometry Index29.428.8
Released2024-06-242024-04-18
WeightsOpenOpen
Context window8K—
Max output2K—
Input $ / M tokens$0.65—
Output $ / M tokens$0.65—
Results tracked3431

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Llama 3-70B leads

Gemma 2 27B: 34.1 (#246), Llama 3-70B: 35.8 (#218)

BenchmarkGemma 2 27BLlama 3-70B
BigCodeBench Instruct42.8%43.6%
LMArena Coding12111206
BigCodeBench Complete52.5%54.5%
LiveBench Coding36%—
HumanEval+—72%
MBPP+—69%

Agentic & Tool Use Not comparable

Gemma 2 27B: —, Llama 3-70B: 21.1 (#139)

BenchmarkGemma 2 27BLlama 3-70B
Cybench—5%

Reasoning Llama 3-70B leads

Gemma 2 27B: 15.3 (#315), Llama 3-70B: 18.0 (#288)

BenchmarkGemma 2 27BLlama 3-70B
LMArena Hard Prompts11981195
DTBench48%54.2%
Epoch Capabilities Index122.08122.93
Kagi LLM Benchmark—35.1%
LiveBench Reasoning28.1%—
LiveBench Data Analysis47.9%—
LMCA7.1%—
ForecastBench—57.1
LiveBench38.2%—
WinoGrande—83.5%

Math Llama 3-70B leads

Gemma 2 27B: 10.7 (#311), Llama 3-70B: 12.8 (#305)

BenchmarkGemma 2 27BLlama 3-70B
OTIS Mock AIME 2024-20251.4%4.3%
LMArena Math12121218
MATH Level 527.9%22.6%
LiveBench Math26.5%—

Knowledge Llama 3-70B leads

Gemma 2 27B: 19.0 (#280), Llama 3-70B: 20.8 (#277)

BenchmarkGemma 2 27BLlama 3-70B
GPQA Diamond36.5%40.6%
LMArena Expert11721149
MMLU75.7%79.3%
Confabulations27.1%—

Multilingual Gemma 2 27B leads

Gemma 2 27B: 38.6 (#226), Llama 3-70B: 33.6 (#251)

BenchmarkGemma 2 27BLlama 3-70B
LMArena Non-English12171142
LMArena Chinese12211114
LMArena French12471232
LMArena German12091169
LMArena Japanese11751017
LMArena Korean11741017
LMArena Russian12341159
LMArena Spanish12281241

Instruction Following Llama 3-70B leads

Gemma 2 27B: 60.5 (#249), Llama 3-70B: 62.5 (#238)

BenchmarkGemma 2 27BLlama 3-70B
LMArena Instruction Following12061194
LiveBench Instruction Following58.1%—

Long Context Gemma 2 27B leads

Gemma 2 27B: 37.3 (#218), Llama 3-70B: 35.6 (#240)

BenchmarkGemma 2 27BLlama 3-70B
LMArena Longer Query12311174

Writing & Preference Gemma 2 27B leads

Gemma 2 27B: 44.2 (#225), Llama 3-70B: 42.8 (#231)

BenchmarkGemma 2 27BLlama 3-70B
LMArena Text12311221
LMArena Creative Writing12411210
LMArena Multi-Turn12241223
LiveBench Language32.6%—

Frequently asked questions

Is Gemma 2 27B better than Llama 3-70B?

Gemma 2 27B and Llama 3-70B score almost the same on the Noometry Index (29.4 vs 28.8), so choose on price, context window or the category you care about most.

Is Gemma 2 27B or Llama 3-70B better for coding?

Llama 3-70B scores higher on coding benchmarks: 35.8 versus 34.1 in the Noometry coding category.

How many benchmarks do Gemma 2 27B and Llama 3-70B share?

25 benchmarks have published results for both models. Gemma 2 27B has 34 scored results on Noometry and Llama 3-70B has 31.

Related comparisons

Go deeper