Noometry

Model comparison

Claude 3.5 Haiku vs Mercury

Mercury is the stronger model overall, scoring 37.6 to 29.2 on the Noometry Index.

Last verified . 8 shared benchmarks.

Claude 3.5 Haiku Anthropic

29.2

Rank #315 Confirmed

Mercury Inception

37.6

Rank #199 Confirmed

Summary

  • They share 8 benchmarks with published results for both. Claude 3.5 Haiku scores higher in 1 category and Mercury in 5 categories; 4 gaps are clear of the uncertainty.
  • The widest gap is in coding, where Mercury leads 38.7 to 32.9.

Side by side

Claude 3.5 HaikuMercury
ProviderAnthropicInception
Noometry Index29.237.6
Released2024-10-22—
WeightsProprietaryProprietary
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked499

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mercury leads

Claude 3.5 Haiku: 32.9 (#265), Mercury: 38.7 (#170)

BenchmarkClaude 3.5 HaikuMercury
LMArena Coding12861322
Aider Polyglot28%—
SciCode27.4%—
WeirdML30.7%—
BigCodeBench Instruct46.1%—
LiveBench Coding51.4%—
BigCodeBench Complete59%—
CadEval32%—

Agentic & Tool Use Not comparable

Claude 3.5 Haiku: 28.0 (#95), Mercury: —

BenchmarkClaude 3.5 HaikuMercury
BALROG19.3%—

Reasoning Too close to call

Claude 3.5 Haiku: 17.7 (#290), Mercury: 17.5 (#293)

BenchmarkClaude 3.5 HaikuMercury
LMArena Hard Prompts12511285
Kagi LLM Benchmark—21.6%
CritPt0%—
LiveBench Reasoning28.1%—
DTBench56.7%—
LiveBench Data Analysis48.5%—
Epoch Capabilities Index127.15—
LiveBench43.5%—

Math Not comparable

Claude 3.5 Haiku: 14.7 (#300), Mercury: —

BenchmarkClaude 3.5 HaikuMercury
OTIS Mock AIME 2024-20254.3%—
Omni-MATH22.4%—
LiveBench Math35.5%—
LMArena Math1244—
MATH Level 546.4%—
FrontierMath (Feb 2025 set)0.3%—

Knowledge Not comparable

Claude 3.5 Haiku: 18.7 (#281), Mercury: —

BenchmarkClaude 3.5 HaikuMercury
GPQA Diamond38.1%—
MMLU-Pro60.5%—
Confabulations36.7%—
GPQA (HELM)36.3%—
LMArena Expert1208—
MMLU74.3%—

Multimodal Not comparable

Claude 3.5 Haiku: 26.8 (#117), Mercury: —

BenchmarkClaude 3.5 HaikuMercury
LMArena Vision1092—
GeoBench34%—

Multilingual Mercury leads

Claude 3.5 Haiku: 40.0 (#218), Mercury: 41.6 (#206)

BenchmarkClaude 3.5 HaikuMercury
LMArena Non-English12381260
LMArena Chinese1229—
LMArena French1264—
LMArena German1237—
LMArena Japanese1175—
LMArena Korean1173—
LMArena Russian1253—
LMArena Spanish1261—

Instruction Following Mercury leads

Claude 3.5 Haiku: 62.9 (#234), Mercury: 65.2 (#224)

BenchmarkClaude 3.5 HaikuMercury
LMArena Instruction Following12411239
LiveBench Instruction Following61.9%—
IFEval79.2%—

Long Context Too close to call

Claude 3.5 Haiku: 38.3 (#200), Mercury: 38.4 (#198)

BenchmarkClaude 3.5 HaikuMercury
LMArena Longer Query12611266

Writing & Preference Mercury leads

Claude 3.5 Haiku: 42.7 (#234), Mercury: 46.2 (#221)

BenchmarkClaude 3.5 HaikuMercury
LMArena Text12551282
LMArena Creative Writing12331191
LMArena Multi-Turn12651282
Short-Story Creative Writing73.5%—
EQ-Bench Creative Writing1146—
WildBench76%—
LiveBench Language35.4%—

Frequently asked questions

Is Claude 3.5 Haiku better than Mercury?

Mercury is the stronger model overall, scoring 37.6 to 29.2 on the Noometry Index.

Is Claude 3.5 Haiku or Mercury better for coding?

Mercury scores higher on coding benchmarks: 38.7 versus 32.9 in the Noometry coding category.

How many benchmarks do Claude 3.5 Haiku and Mercury share?

8 benchmarks have published results for both models. Claude 3.5 Haiku has 49 scored results on Noometry and Mercury has 9.

Related comparisons

Go deeper