Noometry

Model comparison

Grok-2 (Dec 2024) vs o3

o3 is the stronger model overall, scoring 47.5 to 33.7 on the Noometry Index.

Last verified . 27 shared benchmarks.

Grok-2 (Dec 2024) xAI

33.7

Rank #239 Confirmed

o3 OpenAI

47.5

Rank #61 Confirmed

Summary

  • They share 27 benchmarks with published results for both. Grok-2 (Dec 2024) scores higher in 0 categories and o3 in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where o3 leads 50.2 to 20.8.
  • The biggest single-benchmark swing is OTIS Mock AIME 2024-2025: 11.5% for Grok-2 (Dec 2024) and 84.4% for o3.

Side by side

Grok-2 (Dec 2024)o3
ProviderxAIOpenAI
Noometry Index33.747.5
Released2024-08-132025-04-16
WeightsProprietaryProprietary
Context window—200K
Max output—100K
Input $ / M tokens—$2
Output $ / M tokens—$8
Results tracked3463

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o3 leads

Grok-2 (Dec 2024): 33.3 (#258), o3: 46.8 (#64)

BenchmarkGrok-2 (Dec 2024)o3
WeirdML22.2%52.4%
LMArena Coding12871408
SWE-bench Verified—62.3%
SWE-bench Verified (bash only)—58.4%
Aider Polyglot—81.3%
GSO—8.8%
LiveBench Coding46.4%—
CadEval—74%
ALE-Bench—933.55

Agentic & Tool Use Not comparable

Grok-2 (Dec 2024): —, o3: 34.5 (#44)

BenchmarkGrok-2 (Dec 2024)o3
Berkeley Function Calling Leaderboard—63%
GDPval—30.8%
DeepResearch Bench—45.2%
OSWorld—23%
LMArena Search—1144
METR Time Horizons—65.4%

Reasoning o3 leads

Grok-2 (Dec 2024): 16.9 (#299), o3: 32.0 (#78)

BenchmarkGrok-2 (Dec 2024)o3
SimpleBench22.7%53.1%
LMArena Hard Prompts12721402
DTBench65.2%84.8%
Epoch Capabilities Index130.48146.86
ARC-AGI-2—6.5%
Kagi LLM Benchmark—67.6%
ARC-AGI-1—60.8%
CritPt—1.4%
Chess Puzzles—38%
EnigmaEval—13.1%
LiveBench Reasoning54.8%—
Mystery Game Puzzles—29%
LiveBench Data Analysis54.5%—
LMCA—39.7%
ForecastBench—62.5
LiveBench54.3%—

Math o3 leads

Grok-2 (Dec 2024): 20.8 (#284), o3: 50.2 (#58)

BenchmarkGrok-2 (Dec 2024)o3
OTIS Mock AIME 2024-202511.5%84.4%
LMArena Math12831426
MATH Level 563.5%97.8%
FrontierMath (Feb 2025 set)0.7%18.7%
FrontierMath (Tiers 1-3)—33.3%
Omni-MATH—71.4%
LiveBench Math54.9%—
FrontierMath Tier 4 (v1)—2.1%

Knowledge o3 leads

Grok-2 (Dec 2024): 29.8 (#233), o3: 54.6 (#52)

BenchmarkGrok-2 (Dec 2024)o3
GPQA Diamond53.8%81.8%
Confabulations20.1%14.4%
LMArena Expert12541402
Humanity's Last Exam—20.3%
SimpleQA Verified—49.4%
MMLU-Pro—85.9%
GPQA (HELM)—75.3%

Multimodal Not comparable

Grok-2 (Dec 2024): —, o3: 41.4 (#36)

BenchmarkGrok-2 (Dec 2024)o3
LMArena Vision—1214
GeoBench—74%
VPCT—52%

Multilingual o3 leads

Grok-2 (Dec 2024): 43.1 (#188), o3: 51.7 (#105)

BenchmarkGrok-2 (Dec 2024)o3
LMArena Non-English12821401
LMArena Chinese12891437
LMArena French13181430
LMArena German12871420
LMArena Japanese12441403
LMArena Korean12371370
LMArena Russian12861406
LMArena Spanish12811395

Instruction Following o3 leads

Grok-2 (Dec 2024): 66.9 (#202), o3: 72.8 (#127)

BenchmarkGrok-2 (Dec 2024)o3
LMArena Instruction Following12701368
LiveBench Instruction Following69.6%—
IFEval—86.9%

Long Context o3 leads

Grok-2 (Dec 2024): 38.8 (#190), o3: 53.3 (#6)

BenchmarkGrok-2 (Dec 2024)o3
LMArena Longer Query12761372
Fiction.LiveBench—88.9%
CL-bench—17.8%

Writing & Preference o3 leads

Grok-2 (Dec 2024): 48.6 (#198), o3: 63.5 (#64)

BenchmarkGrok-2 (Dec 2024)o3
LMArena Text13051410
LMArena Creative Writing12841359
Short-Story Creative Writing63.6%83.9%
LMArena Multi-Turn12901405
EQ-Bench Creative Writing—1676
WildBench—86.1%
LiveBench Language45.6%—

Frequently asked questions

Is Grok-2 (Dec 2024) better than o3?

o3 is the stronger model overall, scoring 47.5 to 33.7 on the Noometry Index.

Is Grok-2 (Dec 2024) or o3 better for coding?

o3 scores higher on coding benchmarks: 46.8 versus 33.3 in the Noometry coding category.

How many benchmarks do Grok-2 (Dec 2024) and o3 share?

27 benchmarks have published results for both models. Grok-2 (Dec 2024) has 34 scored results on Noometry and o3 has 63.

Related comparisons

Go deeper