Noometry

Model comparison

Claude 3 Sonnet vs o1-mini

o1-mini is the stronger model overall, scoring 33.7 to 28.8 on the Noometry Index.

Last verified . 24 shared benchmarks.

Claude 3 Sonnet Anthropic

28.8

Rank #311 Confirmed

o1-mini OpenAI

33.7

Rank #230 Confirmed

Summary

  • They share 24 benchmarks with published results for both. Claude 3 Sonnet scores higher in 1 category and o1-mini in 7 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where o1-mini leads 33.7 to 10.0.
  • The biggest single-benchmark swing is MATH Level 5: 18.2% for Claude 3 Sonnet and 89.2% for o1-mini.

Side by side

Claude 3 Sonneto1-mini
ProviderAnthropicOpenAI
Noometry Index28.833.7
Released2024-02-292024-09-12
WeightsProprietaryProprietary
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked3039

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o1-mini leads

Claude 3 Sonnet: 29.2 (#296), o1-mini: 35.1 (#218)

BenchmarkClaude 3 Sonneto1-mini
WeirdML10.2%36.3%
LMArena Coding12231362
HumanEval+64%89%
MBPP+69.3%78.8%
Aider Polyglot—32.9%
BigCodeBench Instruct42.7%—
LiveBench Coding—48%
BigCodeBench Complete53.8%—

Agentic & Tool Use Not comparable

Claude 3 Sonnet: —, o1-mini: 24.3 (#112)

BenchmarkClaude 3 Sonneto1-mini
Cybench—10%

Reasoning Claude 3 Sonnet leads

Claude 3 Sonnet: 19.7 (#232), o1-mini: 8.9 (#340)

BenchmarkClaude 3 Sonneto1-mini
LMArena Hard Prompts11971333
Epoch Capabilities Index120.69135.82
ARC-AGI-2—0.8%
SimpleBench—18.1%
ARC-AGI-1—14%
LiveBench Reasoning—72.3%
DTBench53.6%—
LiveBench Data Analysis—57.9%
LiveBench—57.8%
WinoGrande75.1%—

Math o1-mini leads

Claude 3 Sonnet: 10.0 (#305), o1-mini: 33.7 (#168)

BenchmarkClaude 3 Sonneto1-mini
OTIS Mock AIME 2024-20252.5%46.9%
LMArena Math12131358
MATH Level 518.2%89.2%
LiveBench Math—62%
FrontierMath (Feb 2025 set)—1.7%

Knowledge o1-mini leads

Claude 3 Sonnet: 21.1 (#271), o1-mini: 34.8 (#180)

BenchmarkClaude 3 Sonneto1-mini
GPQA Diamond40.6%62.4%
LMArena Expert11731316
Confabulations—18.6%
MMLU75.9%—

Multimodal Not comparable

Claude 3 Sonnet: 24.7 (#120), o1-mini: —

BenchmarkClaude 3 Sonneto1-mini
LMArena Vision984—

Multilingual o1-mini leads

Claude 3 Sonnet: 38.0 (#229), o1-mini: 43.8 (#177)

BenchmarkClaude 3 Sonneto1-mini
LMArena Non-English12051289
LMArena Chinese11881314
LMArena French12281292
LMArena German12041278
LMArena Japanese11311245
LMArena Korean11281223
LMArena Russian12271283
LMArena Spanish12041302

Instruction Following o1-mini leads

Claude 3 Sonnet: 63.1 (#230), o1-mini: 67.0 (#200)

BenchmarkClaude 3 Sonneto1-mini
LMArena Instruction Following12001304
LiveBench Instruction Following—65.4%

Long Context o1-mini leads

Claude 3 Sonnet: 36.7 (#223), o1-mini: 40.1 (#156)

BenchmarkClaude 3 Sonneto1-mini
LMArena Longer Query12111321

Writing & Preference o1-mini leads

Claude 3 Sonnet: 42.3 (#233), o1-mini: 48.6 (#196)

BenchmarkClaude 3 Sonneto1-mini
LMArena Text12181317
LMArena Creative Writing11861244
LMArena Multi-Turn12271314
Short-Story Creative Writing—64.9%
LiveBench Language—40.9%

Frequently asked questions

Is Claude 3 Sonnet better than o1-mini?

o1-mini is the stronger model overall, scoring 33.7 to 28.8 on the Noometry Index.

Is Claude 3 Sonnet or o1-mini better for coding?

o1-mini scores higher on coding benchmarks: 35.1 versus 29.2 in the Noometry coding category.

How many benchmarks do Claude 3 Sonnet and o1-mini share?

24 benchmarks have published results for both models. Claude 3 Sonnet has 30 scored results on Noometry and o1-mini has 39.

Related comparisons

Go deeper