Noometry

Model comparison

Claude 3 Haiku vs o1-mini

o1-mini is the stronger model overall, scoring 34.0 to 25.9 on the Noometry Index.

Last verified . 25 shared benchmarks.

Claude 3 Haiku Anthropic

25.9

Rank #340 Confirmed

o1-mini OpenAI

34.0

Rank #235 Confirmed

Summary

  • They share 25 benchmarks with published results for both. Claude 3 Haiku scores higher in 1 category and o1-mini in 7 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where o1-mini leads 35.4 to 9.8.
  • The biggest single-benchmark swing is MATH Level 5: 14.9% for Claude 3 Haiku and 89.2% for o1-mini.

Side by side

Claude 3 Haikuo1-mini
ProviderAnthropicOpenAI
Noometry Index25.934.0
Released2024-03-072024-09-12
WeightsProprietaryProprietary
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked3739

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o1-mini leads

Claude 3 Haiku: 26.4 (#325), o1-mini: 35.5 (#224)

BenchmarkClaude 3 Haikuo1-mini
WeirdML9.8%36.3%
LMArena Coding11991362
HumanEval+68.9%89%
MBPP+68.8%78.8%
Aider Polyglot—32.9%
BigCodeBench Instruct39.4%—
LiveBench Coding—48%
BigCodeBench Complete50.1%—
CadEval12%—

Agentic & Tool Use Not comparable

Claude 3 Haiku: —, o1-mini: 24.6 (#118)

BenchmarkClaude 3 Haikuo1-mini
Cybench—10%

Reasoning Claude 3 Haiku leads

Claude 3 Haiku: 16.3 (#307), o1-mini: 8.8 (#346)

BenchmarkClaude 3 Haikuo1-mini
LMArena Hard Prompts11741333
Epoch Capabilities Index118.35135.82
ARC-AGI-2—0.8%
SimpleBench—18.1%
Kagi LLM Benchmark34.2%—
ARC-AGI-1—14%
LiveBench Reasoning—72.3%
DTBench50.1%—
LiveBench Data Analysis—57.9%
LMCA8.8%—
ForecastBench53.2—
LiveBench—57.8%
WinoGrande74.2%—

Math o1-mini leads

Claude 3 Haiku: 9.8 (#319), o1-mini: 35.4 (#186)

BenchmarkClaude 3 Haikuo1-mini
OTIS Mock AIME 2024-20251.8%46.9%
LMArena Math11881358
MATH Level 514.9%89.2%
LiveBench Math—62%
FrontierMath (Feb 2025 set)—1.7%

Knowledge o1-mini leads

Claude 3 Haiku: 17.3 (#285), o1-mini: 34.9 (#192)

BenchmarkClaude 3 Haikuo1-mini
GPQA Diamond36.3%62.4%
Confabulations34.2%18.6%
LMArena Expert11481316
MMLU73.8%—

Multimodal Not comparable

Claude 3 Haiku: 23.6 (#128), o1-mini: —

BenchmarkClaude 3 Haikuo1-mini
LMArena Vision950—
ScienceQA72%—

Multilingual o1-mini leads

Claude 3 Haiku: 36.0 (#243), o1-mini: 43.6 (#182)

BenchmarkClaude 3 Haikuo1-mini
LMArena Non-English11781289
LMArena Chinese11551314
LMArena French11951293
LMArena German11741278
LMArena Japanese11021245
LMArena Korean11091223
LMArena Russian12041283
LMArena Spanish11661303

Instruction Following o1-mini leads

Claude 3 Haiku: 61.3 (#247), o1-mini: 66.7 (#206)

BenchmarkClaude 3 Haikuo1-mini
LMArena Instruction Following11731304
LiveBench Instruction Following—65.4%

Long Context o1-mini leads

Claude 3 Haiku: 36.1 (#237), o1-mini: 40.1 (#161)

BenchmarkClaude 3 Haikuo1-mini
LMArena Longer Query11901320

Writing & Preference o1-mini leads

Claude 3 Haiku: 29.7 (#291), o1-mini: 48.4 (#202)

BenchmarkClaude 3 Haikuo1-mini
LMArena Text11951317
LMArena Creative Writing11571244
LMArena Multi-Turn11901314
Short-Story Creative Writing—64.9%
EQ-Bench Creative Writing717—
LiveBench Language—40.9%

Frequently asked questions

Is Claude 3 Haiku better than o1-mini?

o1-mini is the stronger model overall, scoring 34.0 to 25.9 on the Noometry Index.

Is Claude 3 Haiku or o1-mini better for coding?

o1-mini scores higher on coding benchmarks: 35.5 versus 26.4 in the Noometry coding category.

How many benchmarks do Claude 3 Haiku and o1-mini share?

25 benchmarks have published results for both models. Claude 3 Haiku has 37 scored results on Noometry and o1-mini has 39.

Related comparisons

Go deeper