Noometry

Model comparison

Claude 3 Sonnet vs o1

o1 is the stronger model overall, scoring 40.9 to 29.0 on the Noometry Index.

Last verified . 26 shared benchmarks.

Claude 3 Sonnet Anthropic

29.0

Rank #319 Confirmed

o1 OpenAI

40.9

Rank #143 Confirmed

Summary

  • They share 26 benchmarks with published results for both. Claude 3 Sonnet scores higher in 0 categories and o1 in 9 categories; 9 gaps are clear of the uncertainty.
  • The widest gap is in math, where o1 leads 36.1 to 10.7.
  • The biggest single-benchmark swing is MATH Level 5: 18.2% for Claude 3 Sonnet and 94.7% for o1.

Side by side

Claude 3 Sonneto1
ProviderAnthropicOpenAI
Noometry Index29.040.9
Released2024-02-292024-09-12
WeightsProprietaryProprietary
Context window—200K
Max output—100K
Input $ / M tokens—$15
Output $ / M tokens—$60
Results tracked3052

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o1 leads

Claude 3 Sonnet: 29.6 (#302), o1: 46.1 (#70)

BenchmarkClaude 3 Sonneto1
WeirdML10.2%47.6%
LMArena Coding12231367
HumanEval+64%89%
MBPP+69.3%80.2%
Aider Polyglot—61.7%
BigCodeBench Instruct42.7%—
LiveBench Coding—69.7%
BigCodeBench Complete53.8%—
CadEval—56%

Agentic & Tool Use Not comparable

Claude 3 Sonnet: —, o1: 24.6 (#117)

BenchmarkClaude 3 Sonneto1
Cybench—10%
METR Time Horizons—51.1%

Reasoning o1 leads

Claude 3 Sonnet: 20.5 (#237), o1: 27.9 (#111)

BenchmarkClaude 3 Sonneto1
LMArena Hard Prompts11971371
DTBench53.6%74.7%
Epoch Capabilities Index120.7141.91
SimpleBench—41.7%
ARC-AGI-1—30.7%
Chess Puzzles—15%
EnigmaEval—5.7%
LiveBench Reasoning—91.6%
LiveBench Data Analysis—65.5%
LMCA—22.3%
LiveBench—75.7%
WinoGrande75.1%—

Math o1 leads

Claude 3 Sonnet: 10.7 (#310), o1: 36.1 (#175)

BenchmarkClaude 3 Sonneto1
OTIS Mock AIME 2024-20252.5%73.3%
LMArena Math12131388
MATH Level 518.2%94.7%
FrontierMath (Tiers 1-3)—14.7%
LiveBench Math—80.3%
FrontierMath (Feb 2025 set)—9.3%

Knowledge o1 leads

Claude 3 Sonnet: 21.1 (#276), o1: 41.5 (#110)

BenchmarkClaude 3 Sonneto1
GPQA Diamond40.6%76.8%
LMArena Expert11731361
Humanity's Last Exam—8%
SimpleQA Verified—41.1%
Confabulations—11.7%
MMLU75.9%—

Multimodal o1 leads

Claude 3 Sonnet: 25.2 (#125), o1: 34.2 (#93)

BenchmarkClaude 3 Sonneto1
LMArena Vision9841168
GeoBench—80%
VPCT—37%
SpatialViz-Bench—41.4%

Multilingual o1 leads

Claude 3 Sonnet: 37.8 (#234), o1: 48.6 (#142)

BenchmarkClaude 3 Sonneto1
LMArena Non-English12051358
LMArena Chinese11891394
LMArena French12291344
LMArena German12041337
LMArena Japanese11311346
LMArena Korean11281396
LMArena Russian12271356
LMArena Spanish12041345

Instruction Following o1 leads

Claude 3 Sonnet: 62.8 (#235), o1: 74.8 (#86)

BenchmarkClaude 3 Sonneto1
LMArena Instruction Following11991367
LiveBench Instruction Following—81.5%

Long Context o1 leads

Claude 3 Sonnet: 36.7 (#228), o1: 50.3 (#9)

BenchmarkClaude 3 Sonneto1
LMArena Longer Query12111378
Fiction.LiveBench—83.3%

Writing & Preference o1 leads

Claude 3 Sonnet: 42.1 (#238), o1: 55.6 (#144)

BenchmarkClaude 3 Sonneto1
LMArena Text12181366
LMArena Creative Writing11861348
LMArena Multi-Turn12271369
Short-Story Creative Writing—70.2%
LiveBench Language—65.4%

Frequently asked questions

Is Claude 3 Sonnet better than o1?

o1 is the stronger model overall, scoring 40.9 to 29.0 on the Noometry Index.

Is Claude 3 Sonnet or o1 better for coding?

o1 scores higher on coding benchmarks: 46.1 versus 29.6 in the Noometry coding category.

How many benchmarks do Claude 3 Sonnet and o1 share?

26 benchmarks have published results for both models. Claude 3 Sonnet has 30 scored results on Noometry and o1 has 52.

Related comparisons

Go deeper