Noometry

Model comparison

Grok-2 (Dec 2024) vs o1

o1 is the stronger model overall, scoring 40.9 to 33.7 on the Noometry Index.

Last verified . 34 shared benchmarks.

Grok-2 (Dec 2024) xAI

33.7

Rank #239 Confirmed

o1 OpenAI

40.9

Rank #143 Confirmed

Summary

  • They share 34 benchmarks with published results for both. Grok-2 (Dec 2024) scores higher in 0 categories and o1 in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where o1 leads 36.1 to 20.8.
  • The biggest single-benchmark swing is OTIS Mock AIME 2024-2025: 11.5% for Grok-2 (Dec 2024) and 73.3% for o1.

Side by side

Grok-2 (Dec 2024)o1
ProviderxAIOpenAI
Noometry Index33.740.9
Released2024-08-132024-09-12
WeightsProprietaryProprietary
Context window—200K
Max output—100K
Input $ / M tokens—$15
Output $ / M tokens—$60
Results tracked3452

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o1 leads

Grok-2 (Dec 2024): 33.3 (#258), o1: 46.1 (#70)

BenchmarkGrok-2 (Dec 2024)o1
WeirdML22.2%47.6%
LiveBench Coding46.4%69.7%
LMArena Coding12871367
Aider Polyglot—61.7%
CadEval—56%
HumanEval+—89%
MBPP+—80.2%

Agentic & Tool Use Not comparable

Grok-2 (Dec 2024): —, o1: 24.6 (#117)

BenchmarkGrok-2 (Dec 2024)o1
Cybench—10%
METR Time Horizons—51.1%

Reasoning o1 leads

Grok-2 (Dec 2024): 16.9 (#299), o1: 27.9 (#111)

BenchmarkGrok-2 (Dec 2024)o1
SimpleBench22.7%41.7%
LiveBench Reasoning54.8%91.6%
LMArena Hard Prompts12721371
DTBench65.2%74.7%
LiveBench Data Analysis54.5%65.5%
Epoch Capabilities Index130.48141.91
LiveBench54.3%75.7%
ARC-AGI-1—30.7%
Chess Puzzles—15%
EnigmaEval—5.7%
LMCA—22.3%

Math o1 leads

Grok-2 (Dec 2024): 20.8 (#284), o1: 36.1 (#175)

BenchmarkGrok-2 (Dec 2024)o1
OTIS Mock AIME 2024-202511.5%73.3%
LiveBench Math54.9%80.3%
LMArena Math12831388
MATH Level 563.5%94.7%
FrontierMath (Feb 2025 set)0.7%9.3%
FrontierMath (Tiers 1-3)—14.7%

Knowledge o1 leads

Grok-2 (Dec 2024): 29.8 (#233), o1: 41.5 (#110)

BenchmarkGrok-2 (Dec 2024)o1
GPQA Diamond53.8%76.8%
Confabulations20.1%11.7%
LMArena Expert12541361
Humanity's Last Exam—8%
SimpleQA Verified—41.1%

Multimodal Not comparable

Grok-2 (Dec 2024): —, o1: 34.2 (#93)

BenchmarkGrok-2 (Dec 2024)o1
LMArena Vision—1168
GeoBench—80%
VPCT—37%
SpatialViz-Bench—41.4%

Multilingual o1 leads

Grok-2 (Dec 2024): 43.1 (#188), o1: 48.6 (#142)

BenchmarkGrok-2 (Dec 2024)o1
LMArena Non-English12821358
LMArena Chinese12891394
LMArena French13181344
LMArena German12871337
LMArena Japanese12441346
LMArena Korean12371396
LMArena Russian12861356
LMArena Spanish12811345

Instruction Following o1 leads

Grok-2 (Dec 2024): 66.9 (#202), o1: 74.8 (#86)

BenchmarkGrok-2 (Dec 2024)o1
LiveBench Instruction Following69.6%81.5%
LMArena Instruction Following12701367

Long Context o1 leads

Grok-2 (Dec 2024): 38.8 (#190), o1: 50.3 (#9)

BenchmarkGrok-2 (Dec 2024)o1
LMArena Longer Query12761378
Fiction.LiveBench—83.3%

Writing & Preference o1 leads

Grok-2 (Dec 2024): 48.6 (#198), o1: 55.6 (#144)

BenchmarkGrok-2 (Dec 2024)o1
LMArena Text13051366
LMArena Creative Writing12841348
Short-Story Creative Writing63.6%70.2%
LMArena Multi-Turn12901369
LiveBench Language45.6%65.4%

Frequently asked questions

Is Grok-2 (Dec 2024) better than o1?

o1 is the stronger model overall, scoring 40.9 to 33.7 on the Noometry Index.

Is Grok-2 (Dec 2024) or o1 better for coding?

o1 scores higher on coding benchmarks: 46.1 versus 33.3 in the Noometry coding category.

How many benchmarks do Grok-2 (Dec 2024) and o1 share?

34 benchmarks have published results for both models. Grok-2 (Dec 2024) has 34 scored results on Noometry and o1 has 52.

Related comparisons

Go deeper