Noometry

Model comparison

Gemma 2 9B vs o3

o3 is the stronger model overall, scoring 47.4 to 25.7 on the Noometry Index.

Last verified . 22 shared benchmarks.

Gemma 2 9B Google

25.7

Rank #334 Confirmed

o3 OpenAI

47.4

Rank #53 Confirmed

Summary

  • They share 22 benchmarks with published results for both. Gemma 2 9B scores higher in 0 categories and o3 in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where o3 leads 54.5 to 9.7.
  • The biggest single-benchmark swing is OTIS Mock AIME 2024-2025: 0.6% for Gemma 2 9B and 84.4% for o3.
  • Gemma 2 9B has downloadable open weights; the other is API-only.

Side by side

Gemma 2 9Bo3
ProviderGoogleOpenAI
Noometry Index25.747.4
Released2024-06-242025-04-16
WeightsOpenProprietary
Context window—200K
Max output—100K
Input $ / M tokens—$2
Output $ / M tokens—$8
Results tracked3563

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o3 leads

Gemma 2 9B: 29.0 (#299), o3: 47.3 (#57)

BenchmarkGemma 2 9Bo3
LMArena Coding11741408
SWE-bench Verified—62.3%
SWE-bench Verified (bash only)—58.4%
Aider Polyglot—81.3%
GSO—8.8%
WeirdML—52.4%
BigCodeBench Instruct34.7%—
LiveBench Coding22.5%—
BigCodeBench Complete40.6%—
CadEval—74%
ALE-Bench—933.55

Agentic & Tool Use Not comparable

Gemma 2 9B: —, o3: 34.3 (#42)

BenchmarkGemma 2 9Bo3
Berkeley Function Calling Leaderboard—63%
GDPval—30.8%
DeepResearch Bench—45.2%
OSWorld—23%
LMArena Search—1144
METR Time Horizons—65.4%

Reasoning o3 leads

Gemma 2 9B: 15.3 (#306), o3: 32.1 (#66)

BenchmarkGemma 2 9Bo3
LMArena Hard Prompts11721402
Epoch Capabilities Index119.78146.88
ARC-AGI-2—6.5%
SimpleBench—53.1%
Kagi LLM Benchmark—67.6%
ARC-AGI-1—60.8%
CritPt—1.4%
Chess Puzzles—38%
EnigmaEval—13.1%
LiveBench Reasoning15.2%—
Mystery Game Puzzles—29%
DTBench—84.8%
LiveBench Data Analysis36.4%—
LMCA—39.7%
ForecastBench—62.5
LiveBench28.7%—
PIQA83.7%—

Math o3 leads

Gemma 2 9B: 9.3 (#312), o3: 48.9 (#49)

BenchmarkGemma 2 9Bo3
OTIS Mock AIME 2024-20250.6%84.4%
LMArena Math11831426
MATH Level 521%97.8%
FrontierMath (Tiers 1-3)—33.3%
Omni-MATH—71.4%
LiveBench Math19.8%—
FrontierMath (Feb 2025 set)—18.7%
FrontierMath Tier 4 (v1)—2.1%
GSM8K84.9%—

Knowledge o3 leads

Gemma 2 9B: 9.7 (#300), o3: 54.5 (#47)

BenchmarkGemma 2 9Bo3
GPQA Diamond27.5%81.8%
LMArena Expert11471403
Humanity's Last Exam—20.3%
SimpleQA Verified—49.4%
MMLU-Pro—85.9%
Confabulations—14.4%
GPQA (HELM)—75.3%
BoolQ85.7%—
MMLU72.1%—

Multimodal Not comparable

Gemma 2 9B: —, o3: 40.7 (#35)

BenchmarkGemma 2 9Bo3
LMArena Vision—1214
GeoBench—74%
VPCT—52%

Multilingual o3 leads

Gemma 2 9B: 36.8 (#233), o3: 51.9 (#99)

BenchmarkGemma 2 9Bo3
LMArena Non-English11881402
LMArena Chinese11851438
LMArena French11891430
LMArena German11861420
LMArena Japanese11441404
LMArena Korean11371371
LMArena Russian12001407
LMArena Spanish12001395

Instruction Following o3 leads

Gemma 2 9B: 57.8 (#264), o3: 73.0 (#122)

BenchmarkGemma 2 9Bo3
LMArena Instruction Following11781368
LiveBench Instruction Following52.6%—
IFEval—86.9%

Long Context o3 leads

Gemma 2 9B: 36.3 (#228), o3: 53.2 (#6)

BenchmarkGemma 2 9Bo3
LMArena Longer Query11971372
Fiction.LiveBench—88.9%
CL-bench—17.8%

Writing & Preference o3 leads

Gemma 2 9B: 32.3 (#276), o3: 63.7 (#62)

BenchmarkGemma 2 9Bo3
LMArena Text12071410
LMArena Creative Writing12061359
EQ-Bench Creative Writing8411676
LMArena Multi-Turn11931405
Short-Story Creative Writing—83.9%
WildBench—86.1%
LiveBench Language25.5%—

Frequently asked questions

Is Gemma 2 9B better than o3?

o3 is the stronger model overall, scoring 47.4 to 25.7 on the Noometry Index.

Is Gemma 2 9B or o3 better for coding?

o3 scores higher on coding benchmarks: 47.3 versus 29.0 in the Noometry coding category.

How many benchmarks do Gemma 2 9B and o3 share?

22 benchmarks have published results for both models. Gemma 2 9B has 35 scored results on Noometry and o3 has 63.

Related comparisons

Go deeper