Noometry

Model comparison

Gemini 1.5 Flash 8B vs Mixtral 8x7B

Gemini 1.5 Flash 8B is the stronger model overall, scoring 29.9 to 27.1 on the Noometry Index.

Last verified . 19 shared benchmarks.

Gemini 1.5 Flash 8B Google

29.9

Rank #301 Confirmed

Mixtral 8x7B Mistral AI

27.1

Rank #334 Confirmed

Summary

  • They share 19 benchmarks with published results for both. Gemini 1.5 Flash 8B scores higher in 7 categories and Mixtral 8x7B in 1 category; 8 gaps are clear of the uncertainty.
  • The widest gap is in instruction following, where Gemini 1.5 Flash 8B leads 62.8 to 51.0.
  • Mixtral 8x7B has downloadable open weights; the other is API-only.

Side by side

Gemini 1.5 Flash 8BMixtral 8x7B
ProviderGoogleMistral AI
Noometry Index29.927.1
Released2024-10-032023-12-11
WeightsProprietaryOpen
Context window—32K
Max output—32K
Input $ / M tokens—$0.70
Output $ / M tokens—$0.70
Results tracked2138

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 35.5 (#225), Mixtral 8x7B: 32.8 (#269)

BenchmarkGemini 1.5 Flash 8BMixtral 8x7B
LMArena Coding12181126
HumanEval+—39.6%
MBPP+—49.7%

Reasoning Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 20.0 (#244), Mixtral 8x7B: 18.2 (#285)

BenchmarkGemini 1.5 Flash 8BMixtral 8x7B
LMArena Hard Prompts12091115
DTBench50%49.6%
Adversarial NLI—55.2%
Epoch Capabilities Index—118.47
ForecastBench—56.3
HellaSwag—86.7%
PIQA—83.6%
WinoGrande—77.2%

Math Mixtral 8x7B leads

Gemini 1.5 Flash 8B: 14.2 (#302), Mixtral 8x7B: 18.8 (#289)

BenchmarkGemini 1.5 Flash 8BMixtral 8x7B
LMArena Math12071147
OTIS Mock AIME 2024-20254.6%—
Omni-MATH—10.5%
MATH Level 5—10%
GSM8K—74.4%

Knowledge Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 16.0 (#289), Mixtral 8x7B: 11.0 (#301)

BenchmarkGemini 1.5 Flash 8BMixtral 8x7B
GPQA Diamond33%30.6%
LMArena Expert11851088
MMLU-Pro—33.5%
GPQA (HELM)—29.6%
ARC (AI2) Challenge—87.3%
MMLU—70.6%
OpenBookQA—85.8%
TriviaQA—82.2%

Multimodal Not comparable

Gemini 1.5 Flash 8B: 28.2 (#115), Mixtral 8x7B: —

BenchmarkGemini 1.5 Flash 8BMixtral 8x7B
LMArena Vision1044—

Multilingual Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 38.5 (#229), Mixtral 8x7B: 29.6 (#266)

BenchmarkGemini 1.5 Flash 8BMixtral 8x7B
LMArena Non-English12151077
LMArena Chinese12311055
LMArena French12341166
LMArena German12061114
LMArena Japanese1150931
LMArena Korean1140968
LMArena Russian12361090
LMArena Spanish12121111

Instruction Following Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 62.8 (#236), Mixtral 8x7B: 51.0 (#297)

BenchmarkGemini 1.5 Flash 8BMixtral 8x7B
LMArena Instruction Following11991109
IFEval—57.5%

Long Context Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 37.0 (#225), Mixtral 8x7B: 33.4 (#260)

BenchmarkGemini 1.5 Flash 8BMixtral 8x7B
LMArena Longer Query12191103

Writing & Preference Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 42.8 (#232), Mixtral 8x7B: 34.2 (#270)

BenchmarkGemini 1.5 Flash 8BMixtral 8x7B
LMArena Text12261132
LMArena Creative Writing12181109
LMArena Multi-Turn11851115
WildBench—67.3%

Frequently asked questions

Is Gemini 1.5 Flash 8B better than Mixtral 8x7B?

Gemini 1.5 Flash 8B is the stronger model overall, scoring 29.9 to 27.1 on the Noometry Index.

Is Gemini 1.5 Flash 8B or Mixtral 8x7B better for coding?

Gemini 1.5 Flash 8B scores higher on coding benchmarks: 35.5 versus 32.8 in the Noometry coding category.

How many benchmarks do Gemini 1.5 Flash 8B and Mixtral 8x7B share?

19 benchmarks have published results for both models. Gemini 1.5 Flash 8B has 21 scored results on Noometry and Mixtral 8x7B has 38.

Related comparisons

Go deeper