Noometry

Model comparison

Gemma 2 9B vs GPT-4

GPT-4 is the stronger model overall, scoring 29.1 to 25.9 on the Noometry Index.

Last verified . 26 shared benchmarks.

Gemma 2 9B Google

25.9

Rank #341 Confirmed

GPT-4 OpenAI

29.1

Rank #316 Confirmed

Summary

  • They share 26 benchmarks with published results for both. Gemma 2 9B scores higher in 0 categories and GPT-4 in 8 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where GPT-4 leads 18.4 to 9.7.
  • The biggest single-benchmark swing is BigCodeBench Complete: 40.6% for Gemma 2 9B and 57.2% for GPT-4.
  • Gemma 2 9B has downloadable open weights; the other is API-only.

Side by side

Gemma 2 9BGPT-4
ProviderGoogleOpenAI
Noometry Index25.929.1
Released2024-06-242023-03-14
WeightsOpenProprietary
Context window—8K
Max output—8K
Input $ / M tokens—$30
Output $ / M tokens—$60
Results tracked3538

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding GPT-4 leads

Gemma 2 9B: 29.4 (#304), GPT-4: 31.6 (#283)

BenchmarkGemma 2 9BGPT-4
BigCodeBench Instruct34.7%46%
LMArena Coding11731254
BigCodeBench Complete40.6%57.2%
WeirdML—12.4%
LiveBench Coding22.5%—
HumanEval+—79.3%

Agentic & Tool Use Not comparable

Gemma 2 9B: —, GPT-4: —

BenchmarkGemma 2 9BGPT-4
METR Time Horizons—36.1%

Reasoning GPT-4 leads

Gemma 2 9B: 15.9 (#309), GPT-4: 17.8 (#289)

BenchmarkGemma 2 9BGPT-4
LMArena Hard Prompts11711241
Epoch Capabilities Index119.83125.89
Chess Puzzles—4%
LiveBench Reasoning15.2%—
Mystery Game Puzzles—12%
DTBench—62.7%
LiveBench Data Analysis36.4%—
LMCA—17.1%
BIG-Bench Hard—75.1%
ForecastBench—57.8
HellaSwag—95.3%
LiveBench28.7%—
PIQA83.7%—
WinoGrande—87.5%

Math Too close to call

Gemma 2 9B: 9.9 (#318), GPT-4: 10.8 (#309)

BenchmarkGemma 2 9BGPT-4
OTIS Mock AIME 2024-20250.6%1.1%
LMArena Math11831269
MATH Level 521%23%
GSM8K84.9%92%
LiveBench Math19.8%—

Knowledge GPT-4 leads

Gemma 2 9B: 9.7 (#305), GPT-4: 18.4 (#282)

BenchmarkGemma 2 9BGPT-4
GPQA Diamond27.5%35.7%
LMArena Expert11471211
MMLU72.1%86.4%
BoolQ85.7%—
TriviaQA—84.8%

Multilingual GPT-4 leads

Gemma 2 9B: 36.6 (#238), GPT-4: 40.6 (#215)

BenchmarkGemma 2 9BGPT-4
LMArena Non-English11881246
LMArena Chinese11851242
LMArena French11901283
LMArena German11861251
LMArena Japanese11441209
LMArena Korean11371184
LMArena Russian12001251
LMArena Spanish12001261

Instruction Following GPT-4 leads

Gemma 2 9B: 57.6 (#269), GPT-4: 65.3 (#222)

BenchmarkGemma 2 9BGPT-4
LMArena Instruction Following11781241
LiveBench Instruction Following52.6%—

Long Context GPT-4 leads

Gemma 2 9B: 36.3 (#233), GPT-4: 37.7 (#212)

BenchmarkGemma 2 9BGPT-4
LMArena Longer Query11971244

Writing & Preference GPT-4 leads

Gemma 2 9B: 32.1 (#281), GPT-4: 34.9 (#268)

BenchmarkGemma 2 9BGPT-4
LMArena Text12071263
LMArena Creative Writing12061244
EQ-Bench Creative Writing841752
LMArena Multi-Turn11931257
LiveBench Language25.5%—

Frequently asked questions

Is Gemma 2 9B better than GPT-4?

GPT-4 is the stronger model overall, scoring 29.1 to 25.9 on the Noometry Index.

Is Gemma 2 9B or GPT-4 better for coding?

GPT-4 scores higher on coding benchmarks: 31.6 versus 29.4 in the Noometry coding category.

How many benchmarks do Gemma 2 9B and GPT-4 share?

26 benchmarks have published results for both models. Gemma 2 9B has 35 scored results on Noometry and GPT-4 has 38.

Related comparisons

Go deeper