Noometry

Model comparison

Gemma 2 9B vs GPT-4 Turbo

GPT-4 Turbo is the stronger model overall, scoring 30.3 to 25.7 on the Noometry Index.

Last verified . 24 shared benchmarks.

Gemma 2 9B Google

25.7

Rank #334 Confirmed

GPT-4 Turbo OpenAI

30.3

Rank #281 Confirmed

Summary

  • They share 24 benchmarks with published results for both. Gemma 2 9B scores higher in 2 categories and GPT-4 Turbo in 6 categories; 6 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where GPT-4 Turbo leads 47.9 to 32.3.
  • The biggest single-benchmark swing is MATH Level 5: 21% for Gemma 2 9B and 46.7% for GPT-4 Turbo.
  • Gemma 2 9B has downloadable open weights; the other is API-only.

Side by side

Gemma 2 9BGPT-4 Turbo
ProviderGoogleOpenAI
Noometry Index25.730.3
Released2024-06-242023-11-06
WeightsOpenProprietary
Context window—128K
Max output—4K
Input $ / M tokens—$10
Output $ / M tokens—$30
Results tracked3536

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding GPT-4 Turbo leads

Gemma 2 9B: 29.0 (#299), GPT-4 Turbo: 33.4 (#247)

BenchmarkGemma 2 9BGPT-4 Turbo
BigCodeBench Instruct34.7%48.2%
LMArena Coding11741268
BigCodeBench Complete40.6%58.2%
WeirdML—18%
LiveBench Coding22.5%—
HumanEval+—86.6%
MBPP+—73.3%

Agentic & Tool Use Not comparable

Gemma 2 9B: —, GPT-4 Turbo: —

BenchmarkGemma 2 9BGPT-4 Turbo
METR Time Horizons—36.7%

Reasoning Too close to call

Gemma 2 9B: 15.3 (#306), GPT-4 Turbo: 14.9 (#311)

BenchmarkGemma 2 9BGPT-4 Turbo
LMArena Hard Prompts11721251
Epoch Capabilities Index119.78127.25
SimpleBench—25.1%
Chess Puzzles—6%
LiveBench Reasoning15.2%—
DTBench—61.6%
LiveBench Data Analysis36.4%—
LMCA—9.8%
ForecastBench—59.4
LiveBench28.7%—
PIQA83.7%—

Math Too close to call

Gemma 2 9B: 9.3 (#312), GPT-4 Turbo: 8.6 (#316)

BenchmarkGemma 2 9BGPT-4 Turbo
OTIS Mock AIME 2024-20250.6%6.7%
LMArena Math11831272
MATH Level 521%46.7%
FrontierMath (Tiers 1-3)—0.7%
LiveBench Math19.8%—
GSM8K84.9%—

Knowledge GPT-4 Turbo leads

Gemma 2 9B: 9.7 (#300), GPT-4 Turbo: 24.3 (#263)

BenchmarkGemma 2 9BGPT-4 Turbo
GPQA Diamond27.5%46.6%
LMArena Expert11471223
MMLU72.1%81.3%
Confabulations—28.4%
BoolQ85.7%—

Multimodal Not comparable

Gemma 2 9B: —, GPT-4 Turbo: 30.0 (#104)

BenchmarkGemma 2 9BGPT-4 Turbo
LMArena Vision—1090

Multilingual GPT-4 Turbo leads

Gemma 2 9B: 36.8 (#233), GPT-4 Turbo: 40.7 (#211)

BenchmarkGemma 2 9BGPT-4 Turbo
LMArena Non-English11881245
LMArena Chinese11851242
LMArena French11891275
LMArena German11861259
LMArena Japanese11441194
LMArena Korean11371187
LMArena Russian12001259
LMArena Spanish12001260

Instruction Following GPT-4 Turbo leads

Gemma 2 9B: 57.8 (#264), GPT-4 Turbo: 66.0 (#211)

BenchmarkGemma 2 9BGPT-4 Turbo
LMArena Instruction Following11781250
LiveBench Instruction Following52.6%—

Long Context GPT-4 Turbo leads

Gemma 2 9B: 36.3 (#228), GPT-4 Turbo: 38.0 (#200)

BenchmarkGemma 2 9BGPT-4 Turbo
LMArena Longer Query11971254

Writing & Preference GPT-4 Turbo leads

Gemma 2 9B: 32.3 (#276), GPT-4 Turbo: 47.9 (#201)

BenchmarkGemma 2 9BGPT-4 Turbo
LMArena Text12071272
LMArena Creative Writing12061269
LMArena Multi-Turn11931267
EQ-Bench Creative Writing841—
LiveBench Language25.5%—

Frequently asked questions

Is Gemma 2 9B better than GPT-4 Turbo?

GPT-4 Turbo is the stronger model overall, scoring 30.3 to 25.7 on the Noometry Index.

Is Gemma 2 9B or GPT-4 Turbo better for coding?

GPT-4 Turbo scores higher on coding benchmarks: 33.4 versus 29.0 in the Noometry coding category.

How many benchmarks do Gemma 2 9B and GPT-4 Turbo share?

24 benchmarks have published results for both models. Gemma 2 9B has 35 scored results on Noometry and GPT-4 Turbo has 36.

Related comparisons

Go deeper