Noometry

Model comparison

Claude 3 Haiku vs Mistral Small

Mistral Small is the stronger model overall, scoring 33.4 to 25.9 on the Noometry Index.

Last verified . 27 shared benchmarks.

Claude 3 Haiku Anthropic

25.9

Rank #340 Confirmed

Mistral Small Mistral AI

33.4

Rank #243 Confirmed

Summary

  • They share 27 benchmarks with published results for both. Claude 3 Haiku scores higher in 0 categories and Mistral Small in 9 categories; 9 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Mistral Small leads 52.5 to 29.7.
  • The biggest single-benchmark swing is MATH Level 5: 14.9% for Claude 3 Haiku and 46.8% for Mistral Small.
  • Mistral Small has downloadable open weights; the other is API-only.

Side by side

Claude 3 HaikuMistral Small
ProviderAnthropicMistral AI
Noometry Index25.933.4
Released2024-03-072024-02-26
WeightsProprietaryOpen
Context window—262K
Max output—256K
Input $ / M tokens—$0.15
Output $ / M tokens—$0.60
Results tracked3739

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mistral Small leads

Claude 3 Haiku: 26.4 (#325), Mistral Small: 34.0 (#247)

BenchmarkClaude 3 HaikuMistral Small
BigCodeBench Instruct39.4%36.1%
LMArena Coding11991362
BigCodeBench Complete50.1%46.6%
SciCode—26.5%
WeirdML9.8%—
LiveBench Coding—36.2%
CadEval12%—
ALE-Bench—497.62
HumanEval+68.9%—
MBPP+68.8%—

Agentic & Tool Use Not comparable

Claude 3 Haiku: —, Mistral Small: 28.1 (#93)

BenchmarkClaude 3 HaikuMistral Small
Berkeley Function Calling Leaderboard—37.1%

Reasoning Mistral Small leads

Claude 3 Haiku: 16.3 (#307), Mistral Small: 19.8 (#250)

BenchmarkClaude 3 HaikuMistral Small
Kagi LLM Benchmark34.2%37.8%
LMArena Hard Prompts11741335
DTBench50.1%70.9%
LMCA8.8%20.6%
CritPt—0%
LiveBench Reasoning—44.8%
LiveBench Data Analysis—53.7%
Epoch Capabilities Index118.35—
ForecastBench53.2—
LiveBench—44%
WinoGrande74.2%—

Math Mistral Small leads

Claude 3 Haiku: 9.8 (#319), Mistral Small: 16.4 (#293)

BenchmarkClaude 3 HaikuMistral Small
OTIS Mock AIME 2024-20251.8%5.8%
LMArena Math11881341
MATH Level 514.9%46.8%
LiveBench Math—39.9%

Knowledge Mistral Small leads

Claude 3 Haiku: 17.3 (#285), Mistral Small: 31.0 (#222)

BenchmarkClaude 3 HaikuMistral Small
GPQA Diamond36.3%47.5%
LMArena Expert11481291
MMLU73.8%68.7%
Confabulations34.2%—
Vectara Hallucination Rate—5.1%

Multimodal Mistral Small leads

Claude 3 Haiku: 23.6 (#128), Mistral Small: 33.5 (#96)

BenchmarkClaude 3 HaikuMistral Small
LMArena Vision9501142
ScienceQA72%—

Multilingual Mistral Small leads

Claude 3 Haiku: 36.0 (#243), Mistral Small: 45.5 (#169)

BenchmarkClaude 3 HaikuMistral Small
LMArena Non-English11781315
LMArena Chinese11551340
LMArena French11951337
LMArena German11741340
LMArena Japanese11021275
LMArena Korean11091259
LMArena Russian12041324
LMArena Spanish11661346

Instruction Following Mistral Small leads

Claude 3 Haiku: 61.3 (#247), Mistral Small: 66.4 (#209)

BenchmarkClaude 3 HaikuMistral Small
LMArena Instruction Following11731310
LiveBench Instruction Following—63.7%

Long Context Mistral Small leads

Claude 3 Haiku: 36.1 (#237), Mistral Small: 40.4 (#156)

BenchmarkClaude 3 HaikuMistral Small
LMArena Longer Query11901327

Writing & Preference Mistral Small leads

Claude 3 Haiku: 29.7 (#291), Mistral Small: 52.5 (#171)

BenchmarkClaude 3 HaikuMistral Small
LMArena Text11951338
LMArena Creative Writing11571305
LMArena Multi-Turn11901344
EQ-Bench Creative Writing717—
LiveBench Language—30.5%

Frequently asked questions

Is Claude 3 Haiku better than Mistral Small?

Mistral Small is the stronger model overall, scoring 33.4 to 25.9 on the Noometry Index.

Is Claude 3 Haiku or Mistral Small better for coding?

Mistral Small scores higher on coding benchmarks: 34.0 versus 26.4 in the Noometry coding category.

How many benchmarks do Claude 3 Haiku and Mistral Small share?

27 benchmarks have published results for both models. Claude 3 Haiku has 37 scored results on Noometry and Mistral Small has 39.

Related comparisons

Go deeper