Noometry

Model comparison

Gemma 2 9B vs Mistral Large

Mistral Large is the stronger model overall, scoring 31.6 to 25.7 on the Noometry Index.

Last verified . 32 shared benchmarks.

Gemma 2 9B Google

25.7

Rank #334 Confirmed

Mistral Large Mistral AI

31.6

Rank #258 Confirmed

Summary

  • They share 32 benchmarks with published results for both. Gemma 2 9B scores higher in 0 categories and Mistral Large in 8 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Mistral Large leads 30.0 to 9.7.
  • The biggest single-benchmark swing is MATH Level 5: 21% for Gemma 2 9B and 50.3% for Mistral Large.

Side by side

Gemma 2 9BMistral Large
ProviderGoogleMistral AI
Noometry Index25.731.6
Released2024-06-242024-02-26
WeightsOpenOpen
Context window—131K
Max output—16K
Input $ / M tokens—$2
Output $ / M tokens—$6
Results tracked3551

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mistral Large leads

Gemma 2 9B: 29.0 (#299), Mistral Large: 33.8 (#240)

BenchmarkGemma 2 9BMistral Large
BigCodeBench Instruct34.7%30%
LiveBench Coding22.5%47.1%
LMArena Coding11741277
BigCodeBench Complete40.6%38.3%
SciCode—36.2%
ALE-Bench—264.7
HumanEval+—62.2%
MBPP+—59.5%

Agentic & Tool Use Not comparable

Gemma 2 9B: —, Mistral Large: 28.4 (#83)

BenchmarkGemma 2 9BMistral Large
Berkeley Function Calling Leaderboard—38.4%

Reasoning Too close to call

Gemma 2 9B: 15.3 (#306), Mistral Large: 15.4 (#305)

BenchmarkGemma 2 9BMistral Large
LiveBench Reasoning15.2%43.5%
LMArena Hard Prompts11721257
LiveBench Data Analysis36.4%50.1%
Epoch Capabilities Index119.78128.52
LiveBench28.7%48.4%
SimpleBench—22.5%
CritPt—0%
DTBench—65.1%
LMCA—16.7%
ForecastBench—57.1
PIQA83.7%—

Math Mistral Large leads

Gemma 2 9B: 9.3 (#312), Mistral Large: 17.1 (#285)

BenchmarkGemma 2 9BMistral Large
OTIS Mock AIME 2024-20250.6%8.5%
LiveBench Math19.8%42.5%
LMArena Math11831262
MATH Level 521%50.3%
Omni-MATH—28.1%
FrontierMath (Feb 2025 set)—0.3%
GSM8K84.9%—

Knowledge Mistral Large leads

Gemma 2 9B: 9.7 (#300), Mistral Large: 30.0 (#224)

BenchmarkGemma 2 9BMistral Large
GPQA Diamond27.5%51.3%
LMArena Expert11471232
MMLU72.1%80%
MMLU-Pro—59.9%
Confabulations—21.4%
Vectara Hallucination Rate—4.5%
GPQA (HELM)—43.5%
BoolQ85.7%—

Multilingual Mistral Large leads

Gemma 2 9B: 36.8 (#233), Mistral Large: 40.1 (#214)

BenchmarkGemma 2 9BMistral Large
LMArena Non-English11881237
LMArena Chinese11851240
LMArena French11891324
LMArena German11861254
LMArena Japanese11441187
LMArena Korean11371202
LMArena Russian12001257
LMArena Spanish12001268

Instruction Following Mistral Large leads

Gemma 2 9B: 57.8 (#264), Mistral Large: 68.1 (#186)

BenchmarkGemma 2 9BMistral Large
LiveBench Instruction Following52.6%67.9%
LMArena Instruction Following11781249
IFEval—87.7%

Long Context Mistral Large leads

Gemma 2 9B: 36.3 (#228), Mistral Large: 38.3 (#194)

BenchmarkGemma 2 9BMistral Large
LMArena Longer Query11971262

Writing & Preference Mistral Large leads

Gemma 2 9B: 32.3 (#276), Mistral Large: 40.9 (#237)

BenchmarkGemma 2 9BMistral Large
LMArena Text12071266
LMArena Creative Writing12061243
EQ-Bench Creative Writing841985
LMArena Multi-Turn11931260
LiveBench Language25.5%39.4%
Short-Story Creative Writing—69%
WildBench—80.1%

Frequently asked questions

Is Gemma 2 9B better than Mistral Large?

Mistral Large is the stronger model overall, scoring 31.6 to 25.7 on the Noometry Index.

Is Gemma 2 9B or Mistral Large better for coding?

Mistral Large scores higher on coding benchmarks: 33.8 versus 29.0 in the Noometry coding category.

How many benchmarks do Gemma 2 9B and Mistral Large share?

32 benchmarks have published results for both models. Gemma 2 9B has 35 scored results on Noometry and Mistral Large has 51.

Related comparisons

Go deeper