Noometry

Model comparison

Gemma 2 9B vs GPT-5.4 nano

GPT-5.4 nano is the stronger model overall, scoring 41.9 to 25.7 on the Noometry Index.

Last verified . 20 shared benchmarks.

Gemma 2 9B Google

25.7

Rank #334 Confirmed

GPT-5.4 nano OpenAI

41.9

Rank #111 Confirmed

Summary

  • They share 20 benchmarks with published results for both. Gemma 2 9B scores higher in 0 categories and GPT-5.4 nano in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where GPT-5.4 nano leads 41.8 to 9.7.
  • The biggest single-benchmark swing is OTIS Mock AIME 2024-2025: 0.6% for Gemma 2 9B and 87.8% for GPT-5.4 nano.
  • Gemma 2 9B has downloadable open weights; the other is API-only.

Side by side

Gemma 2 9BGPT-5.4 nano
ProviderGoogleOpenAI
Noometry Index25.741.9
Released2024-06-242026-03-17
WeightsOpenProprietary
Context window—400K
Max output—128K
Input $ / M tokens—$0.20
Output $ / M tokens—$1.25
Results tracked3540

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding GPT-5.4 nano leads

Gemma 2 9B: 29.0 (#299), GPT-5.4 nano: 43.1 (#82)

BenchmarkGemma 2 9BGPT-5.4 nano
LMArena Coding11741404
SciCode—46.9%
WeirdML—49.2%
BigCodeBench Instruct34.7%—
LiveBench Coding22.5%—
BigCodeBench Complete40.6%—
ALE-Bench—1,005

Reasoning GPT-5.4 nano leads

Gemma 2 9B: 15.3 (#306), GPT-5.4 nano: 23.8 (#159)

BenchmarkGemma 2 9BGPT-5.4 nano
LMArena Hard Prompts11721380
Epoch Capabilities Index119.78145.78
ARC-AGI-2—5.7%
Kagi LLM Benchmark—39.7%
ARC-AGI-1—51.5%
CritPt—9.3%
Chess Puzzles—30%
LiveBench Reasoning15.2%—
Mystery Game Puzzles—9%
DTBench—80.3%
LiveBench Data Analysis36.4%—
LMCA—36.9%
ForecastBench—57.3
LiveBench28.7%—
PIQA83.7%—

Math GPT-5.4 nano leads

Gemma 2 9B: 9.3 (#312), GPT-5.4 nano: 40.7 (#76)

BenchmarkGemma 2 9BGPT-5.4 nano
OTIS Mock AIME 2024-20250.6%87.8%
LMArena Math11831406
FrontierMath (Tiers 1-3)—44.9%
FrontierMath Tier 4—12.2%
ProofBench—5%
LiveBench Math19.8%—
MATH Level 521%—
FrontierMath (Feb 2025 set)—25.9%
FrontierMath Tier 4 (v1)—6.3%
GSM8K84.9%—

Knowledge GPT-5.4 nano leads

Gemma 2 9B: 9.7 (#300), GPT-5.4 nano: 41.8 (#94)

BenchmarkGemma 2 9BGPT-5.4 nano
GPQA Diamond27.5%78.5%
LMArena Expert11471397
SimpleQA Verified—11.7%
Vectara Hallucination Rate—3.1%
BoolQ85.7%—
MMLU72.1%—

Multimodal Not comparable

Gemma 2 9B: —, GPT-5.4 nano: 36.0 (#73)

BenchmarkGemma 2 9BGPT-5.4 nano
LMArena Vision—1196

Multilingual GPT-5.4 nano leads

Gemma 2 9B: 36.8 (#233), GPT-5.4 nano: 48.8 (#136)

BenchmarkGemma 2 9BGPT-5.4 nano
LMArena Non-English11881359
LMArena Chinese11851391
LMArena French11891396
LMArena German11861368
LMArena Japanese11441342
LMArena Korean11371321
LMArena Russian12001363
LMArena Spanish12001368

Instruction Following GPT-5.4 nano leads

Gemma 2 9B: 57.8 (#264), GPT-5.4 nano: 72.1 (#139)

BenchmarkGemma 2 9BGPT-5.4 nano
LMArena Instruction Following11781362
LiveBench Instruction Following52.6%—

Long Context GPT-5.4 nano leads

Gemma 2 9B: 36.3 (#228), GPT-5.4 nano: 41.6 (#131)

BenchmarkGemma 2 9BGPT-5.4 nano
LMArena Longer Query11971365

Writing & Preference GPT-5.4 nano leads

Gemma 2 9B: 32.3 (#276), GPT-5.4 nano: 55.9 (#137)

BenchmarkGemma 2 9BGPT-5.4 nano
LMArena Text12071372
LMArena Creative Writing12061313
LMArena Multi-Turn11931382
EQ-Bench Creative Writing841—
LiveBench Language25.5%—

Frequently asked questions

Is Gemma 2 9B better than GPT-5.4 nano?

GPT-5.4 nano is the stronger model overall, scoring 41.9 to 25.7 on the Noometry Index.

Is Gemma 2 9B or GPT-5.4 nano better for coding?

GPT-5.4 nano scores higher on coding benchmarks: 43.1 versus 29.0 in the Noometry coding category.

How many benchmarks do Gemma 2 9B and GPT-5.4 nano share?

20 benchmarks have published results for both models. Gemma 2 9B has 35 scored results on Noometry and GPT-5.4 nano has 40.

Related comparisons

Go deeper