Noometry

Model comparison

Claude 3 Sonnet vs Mistral Small 3.1

Mistral Small 3.1 is the stronger model overall, scoring 31.7 to 29.0 on the Noometry Index.

Last verified . 21 shared benchmarks.

Claude 3 Sonnet Anthropic

29.0

Rank #319 Confirmed

Mistral Small 3.1 Mistral AI

31.7

Rank #269 Confirmed

Summary

  • They share 21 benchmarks with published results for both. Claude 3 Sonnet scores higher in 2 categories and Mistral Small 3.1 in 7 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in coding, where Mistral Small 3.1 leads 38.3 to 29.6.
  • Mistral Small 3.1 has downloadable open weights; the other is API-only.

Side by side

Claude 3 SonnetMistral Small 3.1
ProviderAnthropicMistral AI
Noometry Index29.031.7
Released2024-02-292025-03-17
WeightsProprietaryOpen
Context window—128K
Max output—102K
Input $ / M tokens—$0.35
Output $ / M tokens—$0.56
Results tracked3028

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mistral Small 3.1 leads

Claude 3 Sonnet: 29.6 (#302), Mistral Small 3.1: 38.3 (#179)

BenchmarkClaude 3 SonnetMistral Small 3.1
LMArena Coding12231309
WeirdML10.2%—
BigCodeBench Instruct42.7%—
BigCodeBench Complete53.8%—
HumanEval+64%—
MBPP+69.3%—

Reasoning Too close to call

Claude 3 Sonnet: 20.5 (#237), Mistral Small 3.1: 19.7 (#254)

BenchmarkClaude 3 SonnetMistral Small 3.1
LMArena Hard Prompts11971278
Epoch Capabilities Index120.7127.48
Chess Puzzles—1%
DTBench53.6%—
WinoGrande75.1%—

Math Mistral Small 3.1 leads

Claude 3 Sonnet: 10.7 (#310), Mistral Small 3.1: 14.7 (#301)

BenchmarkClaude 3 SonnetMistral Small 3.1
OTIS Mock AIME 2024-20252.5%3.9%
LMArena Math12131262
Omni-MATH—24.8%
MATH Level 518.2%—

Knowledge Mistral Small 3.1 leads

Claude 3 Sonnet: 21.1 (#276), Mistral Small 3.1: 22.6 (#271)

BenchmarkClaude 3 SonnetMistral Small 3.1
GPQA Diamond40.6%41.9%
LMArena Expert11731257
MMLU-Pro—61%
GPQA (HELM)—39.2%
MMLU75.9%—

Multimodal Mistral Small 3.1 leads

Claude 3 Sonnet: 25.2 (#125), Mistral Small 3.1: 33.2 (#99)

BenchmarkClaude 3 SonnetMistral Small 3.1
LMArena Vision9841136

Multilingual Mistral Small 3.1 leads

Claude 3 Sonnet: 37.8 (#234), Mistral Small 3.1: 41.2 (#209)

BenchmarkClaude 3 SonnetMistral Small 3.1
LMArena Non-English12051255
LMArena Chinese11891253
LMArena French12291273
LMArena German12041266
LMArena Japanese11311208
LMArena Korean11281206
LMArena Russian12271263
LMArena Spanish12041283

Instruction Following Too close to call

Claude 3 Sonnet: 62.8 (#235), Mistral Small 3.1: 63.6 (#230)

BenchmarkClaude 3 SonnetMistral Small 3.1
LMArena Instruction Following11991264
IFEval—75%

Long Context Mistral Small 3.1 leads

Claude 3 Sonnet: 36.7 (#228), Mistral Small 3.1: 39.5 (#178)

BenchmarkClaude 3 SonnetMistral Small 3.1
LMArena Longer Query12111299

Writing & Preference Claude 3 Sonnet leads

Claude 3 Sonnet: 42.1 (#238), Mistral Small 3.1: 37.0 (#259)

BenchmarkClaude 3 SonnetMistral Small 3.1
LMArena Text12181277
LMArena Creative Writing11861253
LMArena Multi-Turn12271270
EQ-Bench Creative Writing—761
WildBench—78.8%

Frequently asked questions

Is Claude 3 Sonnet better than Mistral Small 3.1?

Mistral Small 3.1 is the stronger model overall, scoring 31.7 to 29.0 on the Noometry Index.

Is Claude 3 Sonnet or Mistral Small 3.1 better for coding?

Mistral Small 3.1 scores higher on coding benchmarks: 38.3 versus 29.6 in the Noometry coding category.

How many benchmarks do Claude 3 Sonnet and Mistral Small 3.1 share?

21 benchmarks have published results for both models. Claude 3 Sonnet has 30 scored results on Noometry and Mistral Small 3.1 has 28.

Related comparisons

Go deeper