Noometry

Model comparison

Gemma 1.1 7b IT vs Mistral Small

Mistral Small is the stronger model overall, scoring 33.4 to 31.3 on the Noometry Index.

Last verified . 17 shared benchmarks.

Gemma 1.1 7b IT Google

31.3

Rank #277 Confirmed

Mistral Small Mistral AI

33.4

Rank #243 Confirmed

Summary

  • They share 17 benchmarks with published results for both. Gemma 1.1 7b IT scores higher in 2 categories and Mistral Small in 6 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Mistral Small leads 52.5 to 30.4.

Side by side

Gemma 1.1 7b ITMistral Small
ProviderGoogleMistral AI
Noometry Index31.333.4
Released—2024-02-26
WeightsOpenOpen
Context window—262K
Max output—256K
Input $ / M tokens—$0.15
Output $ / M tokens—$0.60
Results tracked1939

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mistral Small leads

Gemma 1.1 7b IT: 31.5 (#284), Mistral Small: 34.0 (#247)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Coding10841362
SciCode—26.5%
BigCodeBench Instruct—36.1%
LiveBench Coding—36.2%
BigCodeBench Complete—46.6%
ALE-Bench—497.62
HumanEval+35.4%—
MBPP+45%—

Agentic & Tool Use Not comparable

Gemma 1.1 7b IT: —, Mistral Small: 28.1 (#93)

BenchmarkGemma 1.1 7b ITMistral Small
Berkeley Function Calling Leaderboard—37.1%

Reasoning Too close to call

Gemma 1.1 7b IT: 20.5 (#238), Mistral Small: 19.8 (#250)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Hard Prompts10711335
Kagi LLM Benchmark—37.8%
CritPt—0%
LiveBench Reasoning—44.8%
DTBench—70.9%
LiveBench Data Analysis—53.7%
LMCA—20.6%
LiveBench—44%

Math Gemma 1.1 7b IT leads

Gemma 1.1 7b IT: 32.0 (#220), Mistral Small: 16.4 (#293)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Math11071341
OTIS Mock AIME 2024-2025—5.8%
LiveBench Math—39.9%
MATH Level 5—46.8%

Knowledge Mistral Small leads

Gemma 1.1 7b IT: 28.3 (#247), Mistral Small: 31.0 (#222)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Expert10391291
GPQA Diamond—47.5%
Vectara Hallucination Rate—5.1%
MMLU—68.7%

Multimodal Not comparable

Gemma 1.1 7b IT: —, Mistral Small: 33.5 (#96)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Vision—1142

Multilingual Mistral Small leads

Gemma 1.1 7b IT: 28.1 (#273), Mistral Small: 45.5 (#169)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Non-English10521315
LMArena Chinese10611340
LMArena French10651337
LMArena German10541340
LMArena Japanese9711275
LMArena Korean9881259
LMArena Russian10461324
LMArena Spanish10491346

Instruction Following Mistral Small leads

Gemma 1.1 7b IT: 54.0 (#283), Mistral Small: 66.4 (#209)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Instruction Following10571310
LiveBench Instruction Following—63.7%

Long Context Mistral Small leads

Gemma 1.1 7b IT: 32.1 (#272), Mistral Small: 40.4 (#156)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Longer Query10561327

Writing & Preference Mistral Small leads

Gemma 1.1 7b IT: 30.4 (#288), Mistral Small: 52.5 (#171)

BenchmarkGemma 1.1 7b ITMistral Small
LMArena Text10941338
LMArena Creative Writing10601305
LMArena Multi-Turn10401344
LiveBench Language—30.5%

Frequently asked questions

Is Gemma 1.1 7b IT better than Mistral Small?

Mistral Small is the stronger model overall, scoring 33.4 to 31.3 on the Noometry Index.

Is Gemma 1.1 7b IT or Mistral Small better for coding?

Mistral Small scores higher on coding benchmarks: 34.0 versus 31.5 in the Noometry coding category.

How many benchmarks do Gemma 1.1 7b IT and Mistral Small share?

17 benchmarks have published results for both models. Gemma 1.1 7b IT has 19 scored results on Noometry and Mistral Small has 39.

Related comparisons

Go deeper