Noometry

Model comparison

Claude 3 Sonnet vs Mistral Small

Mistral Small is the stronger model overall, scoring 33.4 to 29.0 on the Noometry Index.

Last verified . 25 shared benchmarks.

Claude 3 Sonnet Anthropic

29.0

Rank #319 Confirmed

Mistral Small Mistral AI

33.4

Rank #243 Confirmed

Summary

  • They share 25 benchmarks with published results for both. Claude 3 Sonnet scores higher in 1 category and Mistral Small in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Mistral Small leads 52.5 to 42.1.
  • The biggest single-benchmark swing is MATH Level 5: 18.2% for Claude 3 Sonnet and 46.8% for Mistral Small.
  • Mistral Small has downloadable open weights; the other is API-only.

Side by side

Claude 3 SonnetMistral Small
ProviderAnthropicMistral AI
Noometry Index29.033.4
Released2024-02-292024-02-26
WeightsProprietaryOpen
Context window—262K
Max output—256K
Input $ / M tokens—$0.15
Output $ / M tokens—$0.60
Results tracked3039

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mistral Small leads

Claude 3 Sonnet: 29.6 (#302), Mistral Small: 34.0 (#247)

BenchmarkClaude 3 SonnetMistral Small
BigCodeBench Instruct42.7%36.1%
LMArena Coding12231362
BigCodeBench Complete53.8%46.6%
SciCode—26.5%
WeirdML10.2%—
LiveBench Coding—36.2%
ALE-Bench—497.62
HumanEval+64%—
MBPP+69.3%—

Agentic & Tool Use Not comparable

Claude 3 Sonnet: —, Mistral Small: 28.1 (#93)

BenchmarkClaude 3 SonnetMistral Small
Berkeley Function Calling Leaderboard—37.1%

Reasoning Too close to call

Claude 3 Sonnet: 20.5 (#237), Mistral Small: 19.8 (#250)

BenchmarkClaude 3 SonnetMistral Small
LMArena Hard Prompts11971335
DTBench53.6%70.9%
Kagi LLM Benchmark—37.8%
CritPt—0%
LiveBench Reasoning—44.8%
LiveBench Data Analysis—53.7%
LMCA—20.6%
Epoch Capabilities Index120.7—
LiveBench—44%
WinoGrande75.1%—

Math Mistral Small leads

Claude 3 Sonnet: 10.7 (#310), Mistral Small: 16.4 (#293)

BenchmarkClaude 3 SonnetMistral Small
OTIS Mock AIME 2024-20252.5%5.8%
LMArena Math12131341
MATH Level 518.2%46.8%
LiveBench Math—39.9%

Knowledge Mistral Small leads

Claude 3 Sonnet: 21.1 (#276), Mistral Small: 31.0 (#222)

BenchmarkClaude 3 SonnetMistral Small
GPQA Diamond40.6%47.5%
LMArena Expert11731291
MMLU75.9%68.7%
Vectara Hallucination Rate—5.1%

Multimodal Mistral Small leads

Claude 3 Sonnet: 25.2 (#125), Mistral Small: 33.5 (#96)

BenchmarkClaude 3 SonnetMistral Small
LMArena Vision9841142

Multilingual Mistral Small leads

Claude 3 Sonnet: 37.8 (#234), Mistral Small: 45.5 (#169)

BenchmarkClaude 3 SonnetMistral Small
LMArena Non-English12051315
LMArena Chinese11891340
LMArena French12291337
LMArena German12041340
LMArena Japanese11311275
LMArena Korean11281259
LMArena Russian12271324
LMArena Spanish12041346

Instruction Following Mistral Small leads

Claude 3 Sonnet: 62.8 (#235), Mistral Small: 66.4 (#209)

BenchmarkClaude 3 SonnetMistral Small
LMArena Instruction Following11991310
LiveBench Instruction Following—63.7%

Long Context Mistral Small leads

Claude 3 Sonnet: 36.7 (#228), Mistral Small: 40.4 (#156)

BenchmarkClaude 3 SonnetMistral Small
LMArena Longer Query12111327

Writing & Preference Mistral Small leads

Claude 3 Sonnet: 42.1 (#238), Mistral Small: 52.5 (#171)

BenchmarkClaude 3 SonnetMistral Small
LMArena Text12181338
LMArena Creative Writing11861305
LMArena Multi-Turn12271344
LiveBench Language—30.5%

Frequently asked questions

Is Claude 3 Sonnet better than Mistral Small?

Mistral Small is the stronger model overall, scoring 33.4 to 29.0 on the Noometry Index.

Is Claude 3 Sonnet or Mistral Small better for coding?

Mistral Small scores higher on coding benchmarks: 34.0 versus 29.6 in the Noometry coding category.

How many benchmarks do Claude 3 Sonnet and Mistral Small share?

25 benchmarks have published results for both models. Claude 3 Sonnet has 30 scored results on Noometry and Mistral Small has 39.

Related comparisons

Go deeper