Noometry

Model comparison

Claude 3.5 Haiku vs Mercury 2

Mercury 2 is the stronger model overall, scoring 38.8 to 28.9 on the Noometry Index.

Last verified . 14 shared benchmarks.

Claude 3.5 Haiku Anthropic

28.9

Rank #309 Confirmed

Mercury 2 Inception

38.8

Rank #167 Confirmed

Summary

  • They share 14 benchmarks with published results for both. Claude 3.5 Haiku scores higher in 0 categories and Mercury 2 in 7 categories; 6 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Mercury 2 leads 35.6 to 18.7.
  • The biggest single-benchmark swing is WeirdML: 30.7% for Claude 3.5 Haiku and 43.2% for Mercury 2.

Side by side

Claude 3.5 HaikuMercury 2
ProviderAnthropicInception
Noometry Index28.938.8
Released2024-10-222026-02-20
WeightsProprietaryProprietary
Context window—128K
Max output—50K
Input $ / M tokens—$0.25
Output $ / M tokens—$0.75
Results tracked4917

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Too close to call

Claude 3.5 Haiku: 32.6 (#258), Mercury 2: 33.2 (#250)

BenchmarkClaude 3.5 HaikuMercury 2
SciCode27.4%38.7%
WeirdML30.7%43.2%
LMArena Coding12871391
Aider Polyglot28%—
LMArena WebDev—1170
BigCodeBench Instruct46.1%—
LiveBench Coding51.4%—
BigCodeBench Complete59%—
CadEval32%—
ALE-Bench—785.58

Agentic & Tool Use Not comparable

Claude 3.5 Haiku: 27.7 (#90), Mercury 2: —

BenchmarkClaude 3.5 HaikuMercury 2
BALROG19.3%—

Reasoning Mercury 2 leads

Claude 3.5 Haiku: 17.1 (#286), Mercury 2: 23.0 (#170)

BenchmarkClaude 3.5 HaikuMercury 2
CritPt0%0.8%
LMArena Hard Prompts12511362
LiveBench Reasoning28.1%—
DTBench56.7%—
LiveBench Data Analysis48.5%—
Epoch Capabilities Index127.15—
LiveBench43.5%—

Math Not comparable

Claude 3.5 Haiku: 13.7 (#294), Mercury 2: —

BenchmarkClaude 3.5 HaikuMercury 2
OTIS Mock AIME 2024-20254.3%—
Omni-MATH22.4%—
LiveBench Math35.5%—
LMArena Math1244—
MATH Level 546.4%—
FrontierMath (Feb 2025 set)0.3%—

Knowledge Mercury 2 leads

Claude 3.5 Haiku: 18.7 (#276), Mercury 2: 35.6 (#170)

BenchmarkClaude 3.5 HaikuMercury 2
LMArena Expert12081358
GPQA Diamond38.1%—
MMLU-Pro60.5%—
Confabulations36.7%—
Vectara Hallucination Rate—12.3%
GPQA (HELM)36.3%—
MMLU74.3%—

Multimodal Not comparable

Claude 3.5 Haiku: 26.3 (#113), Mercury 2: —

BenchmarkClaude 3.5 HaikuMercury 2
LMArena Vision1092—
GeoBench34%—

Multilingual Mercury 2 leads

Claude 3.5 Haiku: 40.2 (#213), Mercury 2: 46.8 (#153)

BenchmarkClaude 3.5 HaikuMercury 2
LMArena Non-English12381331
LMArena Chinese12291416
LMArena Russian12531303
LMArena French1262—
LMArena German1237—
LMArena Japanese1175—
LMArena Korean1172—
LMArena Spanish1261—

Instruction Following Mercury 2 leads

Claude 3.5 Haiku: 63.1 (#229), Mercury 2: 70.4 (#161)

BenchmarkClaude 3.5 HaikuMercury 2
LMArena Instruction Following12411328
LiveBench Instruction Following61.9%—
IFEval79.2%—

Long Context Mercury 2 leads

Claude 3.5 Haiku: 38.3 (#195), Mercury 2: 40.4 (#149)

BenchmarkClaude 3.5 HaikuMercury 2
LMArena Longer Query12611329

Writing & Preference Mercury 2 leads

Claude 3.5 Haiku: 42.9 (#228), Mercury 2: 54.0 (#151)

BenchmarkClaude 3.5 HaikuMercury 2
LMArena Text12551355
LMArena Creative Writing12331289
LMArena Multi-Turn12651358
Short-Story Creative Writing73.5%—
EQ-Bench Creative Writing1146—
WildBench76%—
LiveBench Language35.4%—

Frequently asked questions

Is Claude 3.5 Haiku better than Mercury 2?

Mercury 2 is the stronger model overall, scoring 38.8 to 28.9 on the Noometry Index.

Is Claude 3.5 Haiku or Mercury 2 better for coding?

They score almost the same on coding (32.6 vs 33.2); test both on your own repository before choosing.

How many benchmarks do Claude 3.5 Haiku and Mercury 2 share?

14 benchmarks have published results for both models. Claude 3.5 Haiku has 49 scored results on Noometry and Mercury 2 has 17.

Related comparisons

Go deeper