Noometry

Model comparison

Gemini 1.5 Flash 8B vs Mistral Small

Mistral Small is the stronger model overall, scoring 33.4 to 29.9 on the Noometry Index.

Last verified . 21 shared benchmarks.

Gemini 1.5 Flash 8B Google

29.9

Rank #301 Confirmed

Mistral Small Mistral AI

33.4

Rank #243 Confirmed

Summary

  • They share 21 benchmarks with published results for both. Gemini 1.5 Flash 8B scores higher in 2 categories and Mistral Small in 7 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Mistral Small leads 31.0 to 16.0.
  • The biggest single-benchmark swing is DTBench: 50% for Gemini 1.5 Flash 8B and 70.9% for Mistral Small.
  • Mistral Small has downloadable open weights; the other is API-only.

Side by side

Gemini 1.5 Flash 8BMistral Small
ProviderGoogleMistral AI
Noometry Index29.933.4
Released2024-10-032024-02-26
WeightsProprietaryOpen
Context window—262K
Max output—256K
Input $ / M tokens—$0.15
Output $ / M tokens—$0.60
Results tracked2139

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 35.5 (#225), Mistral Small: 34.0 (#247)

BenchmarkGemini 1.5 Flash 8BMistral Small
LMArena Coding12181362
SciCode—26.5%
BigCodeBench Instruct—36.1%
LiveBench Coding—36.2%
BigCodeBench Complete—46.6%
ALE-Bench—497.62

Agentic & Tool Use Not comparable

Gemini 1.5 Flash 8B: —, Mistral Small: 28.1 (#93)

BenchmarkGemini 1.5 Flash 8BMistral Small
Berkeley Function Calling Leaderboard—37.1%

Reasoning Too close to call

Gemini 1.5 Flash 8B: 20.0 (#244), Mistral Small: 19.8 (#250)

BenchmarkGemini 1.5 Flash 8BMistral Small
LMArena Hard Prompts12091335
DTBench50%70.9%
Kagi LLM Benchmark—37.8%
CritPt—0%
LiveBench Reasoning—44.8%
LiveBench Data Analysis—53.7%
LMCA—20.6%
LiveBench—44%

Math Mistral Small leads

Gemini 1.5 Flash 8B: 14.2 (#302), Mistral Small: 16.4 (#293)

BenchmarkGemini 1.5 Flash 8BMistral Small
OTIS Mock AIME 2024-20254.6%5.8%
LMArena Math12071341
LiveBench Math—39.9%
MATH Level 5—46.8%

Knowledge Mistral Small leads

Gemini 1.5 Flash 8B: 16.0 (#289), Mistral Small: 31.0 (#222)

BenchmarkGemini 1.5 Flash 8BMistral Small
GPQA Diamond33%47.5%
LMArena Expert11851291
Vectara Hallucination Rate—5.1%
MMLU—68.7%

Multimodal Mistral Small leads

Gemini 1.5 Flash 8B: 28.2 (#115), Mistral Small: 33.5 (#96)

BenchmarkGemini 1.5 Flash 8BMistral Small
LMArena Vision10441142

Multilingual Mistral Small leads

Gemini 1.5 Flash 8B: 38.5 (#229), Mistral Small: 45.5 (#169)

BenchmarkGemini 1.5 Flash 8BMistral Small
LMArena Non-English12151315
LMArena Chinese12311340
LMArena French12341337
LMArena German12061340
LMArena Japanese11501275
LMArena Korean11401259
LMArena Russian12361324
LMArena Spanish12121346

Instruction Following Mistral Small leads

Gemini 1.5 Flash 8B: 62.8 (#236), Mistral Small: 66.4 (#209)

BenchmarkGemini 1.5 Flash 8BMistral Small
LMArena Instruction Following11991310
LiveBench Instruction Following—63.7%

Long Context Mistral Small leads

Gemini 1.5 Flash 8B: 37.0 (#225), Mistral Small: 40.4 (#156)

BenchmarkGemini 1.5 Flash 8BMistral Small
LMArena Longer Query12191327

Writing & Preference Mistral Small leads

Gemini 1.5 Flash 8B: 42.8 (#232), Mistral Small: 52.5 (#171)

BenchmarkGemini 1.5 Flash 8BMistral Small
LMArena Text12261338
LMArena Creative Writing12181305
LMArena Multi-Turn11851344
LiveBench Language—30.5%

Frequently asked questions

Is Gemini 1.5 Flash 8B better than Mistral Small?

Mistral Small is the stronger model overall, scoring 33.4 to 29.9 on the Noometry Index.

Is Gemini 1.5 Flash 8B or Mistral Small better for coding?

Gemini 1.5 Flash 8B scores higher on coding benchmarks: 35.5 versus 34.0 in the Noometry coding category.

How many benchmarks do Gemini 1.5 Flash 8B and Mistral Small share?

21 benchmarks have published results for both models. Gemini 1.5 Flash 8B has 21 scored results on Noometry and Mistral Small has 39.

Related comparisons

Go deeper