Noometry

Model comparison

Gemma 2 9B vs Mixtral 8x22B

Mixtral 8x22B is the stronger model overall, scoring 27.1 to 25.9 on the Noometry Index.

Last verified . 23 shared benchmarks.

Gemma 2 9B Google

25.9

Rank #341 Confirmed

Mixtral 8x22B Mistral AI

27.1

Rank #333 Confirmed

Summary

  • They share 23 benchmarks with published results for both. Gemma 2 9B scores higher in 3 categories and Mixtral 8x22B in 5 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in math, where Mixtral 8x22B leads 22.9 to 9.9.
  • The biggest single-benchmark swing is BigCodeBench Complete: 40.6% for Gemma 2 9B and 50.2% for Mixtral 8x22B.

Side by side

Gemma 2 9BMixtral 8x22B
ProviderGoogleMistral AI
Noometry Index25.927.1
Released2024-06-242024-04-17
WeightsOpenOpen
Context window—64K
Max output—64K
Input $ / M tokens—$2
Output $ / M tokens—$6
Results tracked3534

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemma 2 9B leads

Gemma 2 9B: 29.4 (#304), Mixtral 8x22B: 24.2 (#329)

BenchmarkGemma 2 9BMixtral 8x22B
BigCodeBench Instruct34.7%40.6%
LMArena Coding11731166
BigCodeBench Complete40.6%50.2%
WeirdML—3.2%
LiveBench Coding22.5%—
HumanEval+—72%
MBPP+—64.3%

Agentic & Tool Use Not comparable

Gemma 2 9B: —, Mixtral 8x22B: 23.1 (#127)

BenchmarkGemma 2 9BMixtral 8x22B
Cybench—7.5%

Reasoning Mixtral 8x22B leads

Gemma 2 9B: 15.9 (#309), Mixtral 8x22B: 19.9 (#248)

BenchmarkGemma 2 9BMixtral 8x22B
LMArena Hard Prompts11711150
Epoch Capabilities Index119.83122.03
LiveBench Reasoning15.2%—
DTBench—55.1%
LiveBench Data Analysis36.4%—
ForecastBench—56.3
LiveBench28.7%—
PIQA83.7%—

Math Mixtral 8x22B leads

Gemma 2 9B: 9.9 (#318), Mixtral 8x22B: 22.9 (#275)

BenchmarkGemma 2 9BMixtral 8x22B
LMArena Math11831184
MATH Level 521%24.2%
OTIS Mock AIME 2024-20250.6%—
Omni-MATH—16.3%
LiveBench Math19.8%—
GSM8K84.9%—

Knowledge Mixtral 8x22B leads

Gemma 2 9B: 9.7 (#305), Mixtral 8x22B: 15.1 (#293)

BenchmarkGemma 2 9BMixtral 8x22B
GPQA Diamond27.5%34.1%
LMArena Expert11471113
MMLU72.1%77.8%
MMLU-Pro—46%
GPQA (HELM)—33.4%
BoolQ85.7%—

Multilingual Gemma 2 9B leads

Gemma 2 9B: 36.6 (#238), Mixtral 8x22B: 32.8 (#255)

BenchmarkGemma 2 9BMixtral 8x22B
LMArena Non-English11881128
LMArena Chinese11851116
LMArena French11901166
LMArena German11861141
LMArena Japanese11441037
LMArena Korean11371057
LMArena Russian12001158
LMArena Spanish12001151

Instruction Following Too close to call

Gemma 2 9B: 57.6 (#269), Mixtral 8x22B: 57.7 (#266)

BenchmarkGemma 2 9BMixtral 8x22B
LMArena Instruction Following11781147
LiveBench Instruction Following52.6%—
IFEval—72.4%

Long Context Gemma 2 9B leads

Gemma 2 9B: 36.3 (#233), Mixtral 8x22B: 34.7 (#247)

BenchmarkGemma 2 9BMixtral 8x22B
LMArena Longer Query11971144

Writing & Preference Mixtral 8x22B leads

Gemma 2 9B: 32.1 (#281), Mixtral 8x22B: 36.9 (#262)

BenchmarkGemma 2 9BMixtral 8x22B
LMArena Text12071162
LMArena Creative Writing12061141
LMArena Multi-Turn11931130
EQ-Bench Creative Writing841—
WildBench—71.1%
LiveBench Language25.5%—

Frequently asked questions

Is Gemma 2 9B better than Mixtral 8x22B?

Mixtral 8x22B is the stronger model overall, scoring 27.1 to 25.9 on the Noometry Index.

Is Gemma 2 9B or Mixtral 8x22B better for coding?

Gemma 2 9B scores higher on coding benchmarks: 29.4 versus 24.2 in the Noometry coding category.

How many benchmarks do Gemma 2 9B and Mixtral 8x22B share?

23 benchmarks have published results for both models. Gemma 2 9B has 35 scored results on Noometry and Mixtral 8x22B has 34.

Related comparisons

Go deeper