Noometry

Model comparison

Gemini 1.5 Flash 8B vs Mistral 7B

Gemini 1.5 Flash 8B is the stronger model overall, scoring 29.9 to 23.0 on the Noometry Index.

Last verified . 19 shared benchmarks.

Gemini 1.5 Flash 8B Google

29.9

Rank #301 Confirmed

Mistral 7B Mistral AI

23.0

Rank #351 Confirmed

Summary

  • They share 19 benchmarks with published results for both. Gemini 1.5 Flash 8B scores higher in 8 categories and Mistral 7B in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in multilingual, where Gemini 1.5 Flash 8B leads 38.5 to 25.8.
  • The biggest single-benchmark swing is GPQA Diamond: 33% for Gemini 1.5 Flash 8B and 15.2% for Mistral 7B.
  • Mistral 7B has downloadable open weights; the other is API-only.

Side by side

Gemini 1.5 Flash 8BMistral 7B
ProviderGoogleMistral AI
Noometry Index29.923.0
Released2024-10-032023-09-27
WeightsProprietaryOpen
Context window—8K
Max output—8K
Input $ / M tokens—$0.25
Output $ / M tokens—$0.25
Results tracked2137

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 35.5 (#225), Mistral 7B: 26.4 (#326)

BenchmarkGemini 1.5 Flash 8BMistral 7B
LMArena Coding12181082
BigCodeBench Instruct—19.5%
BigCodeBench Complete—27.3%
HumanEval+—36%
MBPP+—42.1%

Reasoning Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 20.0 (#244), Mistral 7B: 13.1 (#336)

BenchmarkGemini 1.5 Flash 8BMistral 7B
LMArena Hard Prompts12091067
DTBench50%42.5%
Chess Puzzles—0%
Adversarial NLI—47.1%
BIG-Bench Hard—56.1%
Epoch Capabilities Index—112.21
HellaSwag—81%
PIQA—83%
WinoGrande—75.3%

Math Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 14.2 (#302), Mistral 7B: 8.1 (#325)

BenchmarkGemini 1.5 Flash 8BMistral 7B
OTIS Mock AIME 2024-20254.6%0.3%
LMArena Math12071085
MATH Level 5—3.7%
GSM8K—54.4%

Knowledge Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 16.0 (#289), Mistral 7B: 7.4 (#311)

BenchmarkGemini 1.5 Flash 8BMistral 7B
GPQA Diamond33%15.2%
LMArena Expert11851036
ARC (AI2) Challenge—78.6%
BoolQ—87.4%
MMLU—62.5%
OpenBookQA—79.8%
TriviaQA—75.2%

Multimodal Not comparable

Gemini 1.5 Flash 8B: 28.2 (#115), Mistral 7B: —

BenchmarkGemini 1.5 Flash 8BMistral 7B
LMArena Vision1044—

Multilingual Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 38.5 (#229), Mistral 7B: 25.8 (#283)

BenchmarkGemini 1.5 Flash 8BMistral 7B
LMArena Non-English12151012
LMArena Chinese12311009
LMArena French12341037
LMArena German1206987
LMArena Japanese1150878
LMArena Russian12361018
LMArena Spanish12121026
LMArena Korean1140—

Instruction Following Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 62.8 (#236), Mistral 7B: 54.2 (#280)

BenchmarkGemini 1.5 Flash 8BMistral 7B
LMArena Instruction Following11991060

Long Context Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 37.0 (#225), Mistral 7B: 32.2 (#271)

BenchmarkGemini 1.5 Flash 8BMistral 7B
LMArena Longer Query12191060

Writing & Preference Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 42.8 (#232), Mistral 7B: 30.7 (#286)

BenchmarkGemini 1.5 Flash 8BMistral 7B
LMArena Text12261090
LMArena Creative Writing12181068
LMArena Multi-Turn11851062

Frequently asked questions

Is Gemini 1.5 Flash 8B better than Mistral 7B?

Gemini 1.5 Flash 8B is the stronger model overall, scoring 29.9 to 23.0 on the Noometry Index.

Is Gemini 1.5 Flash 8B or Mistral 7B better for coding?

Gemini 1.5 Flash 8B scores higher on coding benchmarks: 35.5 versus 26.4 in the Noometry coding category.

How many benchmarks do Gemini 1.5 Flash 8B and Mistral 7B share?

19 benchmarks have published results for both models. Gemini 1.5 Flash 8B has 21 scored results on Noometry and Mistral 7B has 37.

Related comparisons

Go deeper