Noometry

Model comparison

Claude 3.5 Haiku vs Mistral

Claude 3.5 Haiku and Mistral score almost the same on the Noometry Index (29.2 vs 29.9), so choose on price, context window or the category you care about most.

Last verified . 22 shared benchmarks.

Claude 3.5 Haiku Anthropic

29.2

Rank #315 Confirmed

Mistral Mistral AI

29.9

Rank #303 Confirmed

Summary

  • They share 22 benchmarks with published results for both. Claude 3.5 Haiku scores higher in 5 categories and Mistral in 3 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in instruction following, where Claude 3.5 Haiku leads 62.9 to 52.6.
  • The biggest single-benchmark swing is MMLU-Pro: 60.5% for Claude 3.5 Haiku and 27.7% for Mistral.

Side by side

Claude 3.5 HaikuMistral
ProviderAnthropicMistral AI
Noometry Index29.229.9
Released2024-10-22—
WeightsProprietaryProprietary
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked4922

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Too close to call

Claude 3.5 Haiku: 32.9 (#265), Mistral: 33.8 (#250)

BenchmarkClaude 3.5 HaikuMistral
LMArena Coding12861162
Aider Polyglot28%—
SciCode27.4%—
WeirdML30.7%—
BigCodeBench Instruct46.1%—
LiveBench Coding51.4%—
BigCodeBench Complete59%—
CadEval32%—

Agentic & Tool Use Not comparable

Claude 3.5 Haiku: 28.0 (#95), Mistral: —

BenchmarkClaude 3.5 HaikuMistral
BALROG19.3%—

Reasoning Mistral leads

Claude 3.5 Haiku: 17.7 (#290), Mistral: 22.2 (#200)

BenchmarkClaude 3.5 HaikuMistral
LMArena Hard Prompts12511149
CritPt0%—
LiveBench Reasoning28.1%—
DTBench56.7%—
LiveBench Data Analysis48.5%—
Epoch Capabilities Index127.15—
LiveBench43.5%—

Math Mistral leads

Claude 3.5 Haiku: 14.7 (#300), Mistral: 22.3 (#278)

BenchmarkClaude 3.5 HaikuMistral
Omni-MATH22.4%7.2%
LMArena Math12441180
OTIS Mock AIME 2024-20254.3%—
LiveBench Math35.5%—
MATH Level 546.4%—
FrontierMath (Feb 2025 set)0.3%—

Knowledge Claude 3.5 Haiku leads

Claude 3.5 Haiku: 18.7 (#281), Mistral: 16.6 (#288)

BenchmarkClaude 3.5 HaikuMistral
MMLU-Pro60.5%27.7%
GPQA (HELM)36.3%30.3%
LMArena Expert12081125
GPQA Diamond38.1%—
Confabulations36.7%—
MMLU74.3%—

Multimodal Not comparable

Claude 3.5 Haiku: 26.8 (#117), Mistral: —

BenchmarkClaude 3.5 HaikuMistral
LMArena Vision1092—
GeoBench34%—

Multilingual Claude 3.5 Haiku leads

Claude 3.5 Haiku: 40.0 (#218), Mistral: 32.8 (#254)

BenchmarkClaude 3.5 HaikuMistral
LMArena Non-English12381129
LMArena Chinese12291109
LMArena French12641180
LMArena German12371155
LMArena Japanese11751013
LMArena Korean11731032
LMArena Russian12531168
LMArena Spanish12611143

Instruction Following Claude 3.5 Haiku leads

Claude 3.5 Haiku: 62.9 (#234), Mistral: 52.6 (#288)

BenchmarkClaude 3.5 HaikuMistral
IFEval79.2%56.8%
LMArena Instruction Following12411152
LiveBench Instruction Following61.9%—

Long Context Claude 3.5 Haiku leads

Claude 3.5 Haiku: 38.3 (#200), Mistral: 35.0 (#245)

BenchmarkClaude 3.5 HaikuMistral
LMArena Longer Query12611153

Writing & Preference Claude 3.5 Haiku leads

Claude 3.5 Haiku: 42.7 (#234), Mistral: 37.0 (#260)

BenchmarkClaude 3.5 HaikuMistral
LMArena Text12551165
LMArena Creative Writing12331158
WildBench76%66%
LMArena Multi-Turn12651147
Short-Story Creative Writing73.5%—
EQ-Bench Creative Writing1146—
LiveBench Language35.4%—

Frequently asked questions

Is Claude 3.5 Haiku better than Mistral?

Claude 3.5 Haiku and Mistral score almost the same on the Noometry Index (29.2 vs 29.9), so choose on price, context window or the category you care about most.

Is Claude 3.5 Haiku or Mistral better for coding?

They score almost the same on coding (32.9 vs 33.8); test both on your own repository before choosing.

How many benchmarks do Claude 3.5 Haiku and Mistral share?

22 benchmarks have published results for both models. Claude 3.5 Haiku has 49 scored results on Noometry and Mistral has 22.

Related comparisons

Go deeper