Noometry

Model comparison

Claude 2 vs Mistral Large

Mistral Large is the stronger model overall, scoring 31.9 to 25.0 on the Noometry Index.

Last verified . 7 shared benchmarks.

Claude 2 Anthropic

25.0

Rank #346 Reported

Mistral Large Mistral AI

31.9

Rank #263 Confirmed

Summary

  • They share 7 benchmarks with published results for both. Claude 2 scores higher in 1 category and Mistral Large in 2 categories; 3 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Mistral Large leads 30.1 to 16.9.
  • The biggest single-benchmark swing is MATH Level 5: 11.7% for Claude 2 and 50.3% for Mistral Large.
  • Mistral Large has downloadable open weights; the other is API-only.

Side by side

Claude 2Mistral Large
ProviderAnthropicMistral AI
Noometry Index25.031.9
Released2023-07-112024-02-26
WeightsProprietaryOpen
Context window—131K
Max output—16K
Input $ / M tokens—$2
Output $ / M tokens—$6
Results tracked851

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Claude 2: —, Mistral Large: 34.3 (#240)

BenchmarkClaude 2Mistral Large
HumanEval+61.6%62.2%
SciCode—36.2%
BigCodeBench Instruct—30%
LiveBench Coding—47.1%
LMArena Coding—1277
BigCodeBench Complete—38.3%
ALE-Bench—264.7
MBPP+—59.5%

Agentic & Tool Use Not comparable

Claude 2: —, Mistral Large: 28.6 (#89)

BenchmarkClaude 2Mistral Large
Berkeley Function Calling Leaderboard—38.4%

Reasoning Claude 2 leads

Claude 2: 21.7 (#216), Mistral Large: 15.8 (#310)

BenchmarkClaude 2Mistral Large
DTBench51.9%65.1%
Epoch Capabilities Index120.13128.52
SimpleBench—22.5%
CritPt—0%
LiveBench Reasoning—43.5%
LMArena Hard Prompts—1257
LiveBench Data Analysis—50.1%
LMCA—16.7%
ForecastBench—57.1
LiveBench—48.4%

Math Mistral Large leads

Claude 2: 9.3 (#320), Mistral Large: 18.2 (#291)

BenchmarkClaude 2Mistral Large
OTIS Mock AIME 2024-20252.5%8.5%
MATH Level 511.7%50.3%
Omni-MATH—28.1%
LiveBench Math—42.5%
LMArena Math—1262
FrontierMath (Feb 2025 set)—0.3%

Knowledge Mistral Large leads

Claude 2: 16.9 (#287), Mistral Large: 30.1 (#230)

BenchmarkClaude 2Mistral Large
GPQA Diamond34.7%51.3%
MMLU78.5%80%
MMLU-Pro—59.9%
Confabulations—21.4%
Vectara Hallucination Rate—4.5%
GPQA (HELM)—43.5%
LMArena Expert—1232
TriviaQA87.5%—

Multilingual Not comparable

Claude 2: —, Mistral Large: 40.0 (#219)

BenchmarkClaude 2Mistral Large
LMArena Non-English—1237
LMArena Chinese—1240
LMArena French—1325
LMArena German—1254
LMArena Japanese—1188
LMArena Korean—1202
LMArena Russian—1257
LMArena Spanish—1268

Instruction Following Not comparable

Claude 2: —, Mistral Large: 67.9 (#191)

BenchmarkClaude 2Mistral Large
LiveBench Instruction Following—67.9%
IFEval—87.7%
LMArena Instruction Following—1249

Long Context Not comparable

Claude 2: —, Mistral Large: 38.3 (#199)

BenchmarkClaude 2Mistral Large
LMArena Longer Query—1261

Writing & Preference Not comparable

Claude 2: —, Mistral Large: 40.7 (#242)

BenchmarkClaude 2Mistral Large
LMArena Text—1266
LMArena Creative Writing—1243
Short-Story Creative Writing—69%
EQ-Bench Creative Writing—985
WildBench—80.1%
LMArena Multi-Turn—1260
LiveBench Language—39.4%

Frequently asked questions

Is Claude 2 better than Mistral Large?

Mistral Large is the stronger model overall, scoring 31.9 to 25.0 on the Noometry Index.

How many benchmarks do Claude 2 and Mistral Large share?

7 benchmarks have published results for both models. Claude 2 has 8 scored results on Noometry and Mistral Large has 51.

Related comparisons

Go deeper