Noometry

Model comparison

Claude 3 Haiku vs Ministral 8B

Ministral 8B is the stronger model overall, scoring 27.8 to 25.7 on the Noometry Index.

Last verified . 15 shared benchmarks.

Claude 3 Haiku Anthropic

25.7

Rank #333 Confirmed

Ministral 8B Mistral AI

27.8

Rank #321 Confirmed

Summary

  • They share 15 benchmarks with published results for both. Claude 3 Haiku scores higher in 3 categories and Ministral 8B in 5 categories; 5 gaps are clear of the uncertainty.
  • The widest gap is in math, where Ministral 8B leads 23.7 to 9.2.
  • The biggest single-benchmark swing is GPQA Diamond: 36.3% for Claude 3 Haiku and 27.1% for Ministral 8B.
  • Ministral 8B has downloadable open weights; the other is API-only.

Side by side

Claude 3 HaikuMinistral 8B
ProviderAnthropicMistral AI
Noometry Index25.727.8
Released2024-03-072024-10-01
WeightsProprietaryOpen
Context window—128K
Max output—128K
Input $ / M tokens—$0.10
Output $ / M tokens—$0.10
Results tracked3717

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Ministral 8B leads

Claude 3 Haiku: 26.1 (#320), Ministral 8B: 34.4 (#226)

BenchmarkClaude 3 HaikuMinistral 8B
LMArena Coding11991202
WeirdML9.8%—
BigCodeBench Instruct39.4%—
BigCodeBench Complete50.1%—
CadEval12%—
HumanEval+68.9%—
MBPP+68.8%—

Agentic & Tool Use Not comparable

Claude 3 Haiku: —, Ministral 8B: 16.2 (#140)

BenchmarkClaude 3 HaikuMinistral 8B
Berkeley Function Calling Leaderboard—11.1%

Reasoning Ministral 8B leads

Claude 3 Haiku: 15.7 (#303), Ministral 8B: 17.7 (#279)

BenchmarkClaude 3 HaikuMinistral 8B
LMArena Hard Prompts11741191
DTBench50.1%45.7%
Kagi LLM Benchmark34.2%—
LMCA8.8%—
Epoch Capabilities Index118.3—
ForecastBench53.2—
WinoGrande74.2%—

Math Ministral 8B leads

Claude 3 Haiku: 9.2 (#313), Ministral 8B: 23.7 (#262)

BenchmarkClaude 3 HaikuMinistral 8B
LMArena Math11881188
MATH Level 514.9%14.9%
OTIS Mock AIME 2024-20251.8%—

Knowledge Claude 3 Haiku leads

Claude 3 Haiku: 17.3 (#280), Ministral 8B: 12.5 (#292)

BenchmarkClaude 3 HaikuMinistral 8B
GPQA Diamond36.3%27.1%
LMArena Expert11481170
Confabulations34.2%—
Vectara Hallucination Rate—7.4%
MMLU73.8%—

Multimodal Not comparable

Claude 3 Haiku: 23.1 (#123), Ministral 8B: —

BenchmarkClaude 3 HaikuMinistral 8B
LMArena Vision950—
ScienceQA72%—

Multilingual Too close to call

Claude 3 Haiku: 36.1 (#238), Ministral 8B: 35.3 (#242)

BenchmarkClaude 3 HaikuMinistral 8B
LMArena Non-English11781165
LMArena Chinese11551193
LMArena Russian12041195
LMArena French1194—
LMArena German1174—
LMArena Japanese1102—
LMArena Korean1108—
LMArena Spanish1165—

Instruction Following Too close to call

Claude 3 Haiku: 61.5 (#242), Ministral 8B: 60.7 (#245)

BenchmarkClaude 3 HaikuMinistral 8B
LMArena Instruction Following11741161

Long Context Too close to call

Claude 3 Haiku: 36.1 (#232), Ministral 8B: 36.7 (#222)

BenchmarkClaude 3 HaikuMinistral 8B
LMArena Longer Query11911212

Writing & Preference Ministral 8B leads

Claude 3 Haiku: 29.8 (#286), Ministral 8B: 39.8 (#241)

BenchmarkClaude 3 HaikuMinistral 8B
LMArena Text11951191
LMArena Creative Writing11571175
LMArena Multi-Turn11901166
EQ-Bench Creative Writing717—

Frequently asked questions

Is Claude 3 Haiku better than Ministral 8B?

Ministral 8B is the stronger model overall, scoring 27.8 to 25.7 on the Noometry Index.

Is Claude 3 Haiku or Ministral 8B better for coding?

Ministral 8B scores higher on coding benchmarks: 34.4 versus 26.1 in the Noometry coding category.

How many benchmarks do Claude 3 Haiku and Ministral 8B share?

15 benchmarks have published results for both models. Claude 3 Haiku has 37 scored results on Noometry and Ministral 8B has 17.

Related comparisons

Go deeper