Noometry

Model comparison

Gemma 2 9B vs o3-mini

o3-mini is the stronger model overall, scoring 36.7 to 25.9 on the Noometry Index.

Last verified . 28 shared benchmarks.

Gemma 2 9B Google

25.9

Rank #341 Confirmed

o3-mini OpenAI

36.7

Rank #212 Confirmed

Summary

  • They share 28 benchmarks with published results for both. Gemma 2 9B scores higher in 1 category and o3-mini in 7 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where o3-mini leads 38.3 to 9.7.
  • The biggest single-benchmark swing is OTIS Mock AIME 2024-2025: 0.6% for Gemma 2 9B and 76.9% for o3-mini.
  • Gemma 2 9B has downloadable open weights; the other is API-only.

Side by side

Gemma 2 9Bo3-mini
ProviderGoogleOpenAI
Noometry Index25.936.7
Released2024-06-242024-12-20
WeightsOpenProprietary
Context window—200K
Max output—100K
Input $ / M tokens—$1.10
Output $ / M tokens—$4.40
Results tracked3551

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o3-mini leads

Gemma 2 9B: 29.4 (#304), o3-mini: 40.8 (#132)

BenchmarkGemma 2 9Bo3-mini
LiveBench Coding22.5%82.7%
LMArena Coding11731378
Aider Polyglot—60.4%
SciCode—39.8%
GSO—1.3%
WeirdML—43.7%
BigCodeBench Instruct34.7%—
BigCodeBench Complete40.6%—
CadEval—54%

Agentic & Tool Use Not comparable

Gemma 2 9B: —, o3-mini: 29.6 (#84)

BenchmarkGemma 2 9Bo3-mini
Cybench—22.5%

Reasoning Too close to call

Gemma 2 9B: 15.9 (#309), o3-mini: 16.3 (#305)

BenchmarkGemma 2 9Bo3-mini
LiveBench Reasoning15.2%89.6%
LMArena Hard Prompts11711366
LiveBench Data Analysis36.4%70.6%
Epoch Capabilities Index119.83140.34
LiveBench28.7%75.9%
ARC-AGI-2—3%
SimpleBench—22.8%
ARC-AGI-1—34.5%
CritPt—0.3%
Chess Puzzles—17%
Mystery Game Puzzles—7%
DTBench—68.8%
LMCA—19%
ForecastBench—59.6
PIQA83.7%—

Math o3-mini leads

Gemma 2 9B: 9.9 (#318), o3-mini: 28.1 (#244)

BenchmarkGemma 2 9Bo3-mini
OTIS Mock AIME 2024-20250.6%76.9%
LiveBench Math19.8%77.3%
LMArena Math11831396
MATH Level 521%96.5%
FrontierMath (Tiers 1-3)—18.6%
FrontierMath Tier 4—0%
FrontierMath (Feb 2025 set)—12.4%
FrontierMath Tier 4 (v1)—4.2%
GSM8K84.9%—

Knowledge o3-mini leads

Gemma 2 9B: 9.7 (#305), o3-mini: 38.3 (#146)

BenchmarkGemma 2 9Bo3-mini
GPQA Diamond27.5%77%
LMArena Expert11471364
SimpleQA Verified—15.3%
Confabulations—17.9%
BoolQ85.7%—
MMLU72.1%—

Multilingual o3-mini leads

Gemma 2 9B: 36.6 (#238), o3-mini: 45.7 (#164)

BenchmarkGemma 2 9Bo3-mini
LMArena Non-English11881319
LMArena Chinese11851379
LMArena French11901334
LMArena German11861303
LMArena Japanese11441286
LMArena Korean11371314
LMArena Russian12001304
LMArena Spanish12001321

Instruction Following o3-mini leads

Gemma 2 9B: 57.6 (#269), o3-mini: 75.1 (#72)

BenchmarkGemma 2 9Bo3-mini
LiveBench Instruction Following52.6%84.4%
LMArena Instruction Following11781337

Long Context Gemma 2 9B leads

Gemma 2 9B: 36.3 (#233), o3-mini: 33.8 (#256)

BenchmarkGemma 2 9Bo3-mini
LMArena Longer Query11971343
Fiction.LiveBench—50%

Writing & Preference o3-mini leads

Gemma 2 9B: 32.1 (#281), o3-mini: 50.3 (#182)

BenchmarkGemma 2 9Bo3-mini
LMArena Text12071337
LMArena Creative Writing12061286
LMArena Multi-Turn11931320
LiveBench Language25.5%50.7%
Short-Story Creative Writing—61.7%
EQ-Bench Creative Writing841—

Frequently asked questions

Is Gemma 2 9B better than o3-mini?

o3-mini is the stronger model overall, scoring 36.7 to 25.9 on the Noometry Index.

Is Gemma 2 9B or o3-mini better for coding?

o3-mini scores higher on coding benchmarks: 40.8 versus 29.4 in the Noometry coding category.

How many benchmarks do Gemma 2 9B and o3-mini share?

28 benchmarks have published results for both models. Gemma 2 9B has 35 scored results on Noometry and o3-mini has 51.

Related comparisons

Go deeper