Noometry

Model comparison

Gemma 2 9B vs Llama 3.1 Nemotron Ultra 253b v1

Llama 3.1 Nemotron Ultra 253b v1 is the stronger model overall, scoring 36.7 to 25.9 on the Noometry Index.

Last verified . 10 shared benchmarks.

Gemma 2 9B Google

25.9

Rank #341 Confirmed

Llama 3.1 Nemotron Ultra 253b v1 NVIDIA

36.7

Rank #213 Confirmed

Summary

  • They share 10 benchmarks with published results for both. Gemma 2 9B scores higher in 0 categories and Llama 3.1 Nemotron Ultra 253b v1 in 7 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in math, where Llama 3.1 Nemotron Ultra 253b v1 leads 37.5 to 9.9.

Side by side

Gemma 2 9BLlama 3.1 Nemotron Ultra 253b v1
ProviderGoogleNVIDIA
Noometry Index25.936.7
Released2024-06-24—
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked3511

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Llama 3.1 Nemotron Ultra 253b v1 leads

Gemma 2 9B: 29.4 (#304), Llama 3.1 Nemotron Ultra 253b v1: 38.4 (#177)

BenchmarkGemma 2 9BLlama 3.1 Nemotron Ultra 253b v1
LMArena Coding11731312
BigCodeBench Instruct34.7%—
LiveBench Coding22.5%—
BigCodeBench Complete40.6%—

Agentic & Tool Use Not comparable

Gemma 2 9B: —, Llama 3.1 Nemotron Ultra 253b v1: 15.7 (#149)

BenchmarkGemma 2 9BLlama 3.1 Nemotron Ultra 253b v1
Berkeley Function Calling Leaderboard—10%

Reasoning Llama 3.1 Nemotron Ultra 253b v1 leads

Gemma 2 9B: 15.9 (#309), Llama 3.1 Nemotron Ultra 253b v1: 26.3 (#134)

BenchmarkGemma 2 9BLlama 3.1 Nemotron Ultra 253b v1
LMArena Hard Prompts11711316
LiveBench Reasoning15.2%—
LiveBench Data Analysis36.4%—
Epoch Capabilities Index119.83—
LiveBench28.7%—
PIQA83.7%—

Math Llama 3.1 Nemotron Ultra 253b v1 leads

Gemma 2 9B: 9.9 (#318), Llama 3.1 Nemotron Ultra 253b v1: 37.5 (#152)

BenchmarkGemma 2 9BLlama 3.1 Nemotron Ultra 253b v1
LMArena Math11831360
OTIS Mock AIME 2024-20250.6%—
LiveBench Math19.8%—
MATH Level 521%—
GSM8K84.9%—

Knowledge Not comparable

Gemma 2 9B: 9.7 (#305), Llama 3.1 Nemotron Ultra 253b v1: —

BenchmarkGemma 2 9BLlama 3.1 Nemotron Ultra 253b v1
GPQA Diamond27.5%—
LMArena Expert1147—
BoolQ85.7%—
MMLU72.1%—

Multilingual Llama 3.1 Nemotron Ultra 253b v1 leads

Gemma 2 9B: 36.6 (#238), Llama 3.1 Nemotron Ultra 253b v1: 43.1 (#187)

BenchmarkGemma 2 9BLlama 3.1 Nemotron Ultra 253b v1
LMArena Non-English11881282
LMArena Russian12001284
LMArena Chinese1185—
LMArena French1190—
LMArena German1186—
LMArena Japanese1144—
LMArena Korean1137—
LMArena Spanish1200—

Instruction Following Llama 3.1 Nemotron Ultra 253b v1 leads

Gemma 2 9B: 57.6 (#269), Llama 3.1 Nemotron Ultra 253b v1: 69.0 (#178)

BenchmarkGemma 2 9BLlama 3.1 Nemotron Ultra 253b v1
LMArena Instruction Following11781308
LiveBench Instruction Following52.6%—

Long Context Llama 3.1 Nemotron Ultra 253b v1 leads

Gemma 2 9B: 36.3 (#233), Llama 3.1 Nemotron Ultra 253b v1: 39.5 (#177)

BenchmarkGemma 2 9BLlama 3.1 Nemotron Ultra 253b v1
LMArena Longer Query11971299

Writing & Preference Llama 3.1 Nemotron Ultra 253b v1 leads

Gemma 2 9B: 32.1 (#281), Llama 3.1 Nemotron Ultra 253b v1: 52.2 (#175)

BenchmarkGemma 2 9BLlama 3.1 Nemotron Ultra 253b v1
LMArena Text12071320
LMArena Creative Writing12061314
LMArena Multi-Turn11931317
EQ-Bench Creative Writing841—
LiveBench Language25.5%—

Frequently asked questions

Is Gemma 2 9B better than Llama 3.1 Nemotron Ultra 253b v1?

Llama 3.1 Nemotron Ultra 253b v1 is the stronger model overall, scoring 36.7 to 25.9 on the Noometry Index.

Is Gemma 2 9B or Llama 3.1 Nemotron Ultra 253b v1 better for coding?

Llama 3.1 Nemotron Ultra 253b v1 scores higher on coding benchmarks: 38.4 versus 29.4 in the Noometry coding category.

How many benchmarks do Gemma 2 9B and Llama 3.1 Nemotron Ultra 253b v1 share?

10 benchmarks have published results for both models. Gemma 2 9B has 35 scored results on Noometry and Llama 3.1 Nemotron Ultra 253b v1 has 11.

Related comparisons

Go deeper