Noometry

Model comparison

Granite 3.1 8b Instruct vs Mixtral 8x7B

Granite 3.1 8b Instruct is the stronger model overall, scoring 32.4 to 27.1 on the Noometry Index.

Last verified . 12 shared benchmarks.

Granite 3.1 8b Instruct IBM

32.4

Rank #258 Confirmed

Mixtral 8x7B Mistral AI

27.1

Rank #334 Confirmed

Summary

  • They share 12 benchmarks with published results for both. Granite 3.1 8b Instruct scores higher in 8 categories and Mixtral 8x7B in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Granite 3.1 8b Instruct leads 31.1 to 11.0.

Side by side

Granite 3.1 8b InstructMixtral 8x7B
ProviderIBMMistral AI
Noometry Index32.427.1
Released—2023-12-11
WeightsOpenOpen
Context window—32K
Max output—32K
Input $ / M tokens—$0.70
Output $ / M tokens—$0.70
Results tracked1338

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 34.5 (#233), Mixtral 8x7B: 32.8 (#269)

BenchmarkGranite 3.1 8b InstructMixtral 8x7B
LMArena Coding11861126
HumanEval+—39.6%
MBPP+—49.7%

Agentic & Tool Use Not comparable

Granite 3.1 8b Instruct: 24.1 (#120), Mixtral 8x7B: —

BenchmarkGranite 3.1 8b InstructMixtral 8x7B
Berkeley Function Calling Leaderboard27.1%—

Reasoning Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 22.1 (#207), Mixtral 8x7B: 18.2 (#285)

BenchmarkGranite 3.1 8b InstructMixtral 8x7B
LMArena Hard Prompts11451115
DTBench—49.6%
Adversarial NLI—55.2%
Epoch Capabilities Index—118.47
ForecastBench—56.3
HellaSwag—86.7%
PIQA—83.6%
WinoGrande—77.2%

Math Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 33.0 (#209), Mixtral 8x7B: 18.8 (#289)

BenchmarkGranite 3.1 8b InstructMixtral 8x7B
LMArena Math11521147
Omni-MATH—10.5%
MATH Level 5—10%
GSM8K—74.4%

Knowledge Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 31.1 (#220), Mixtral 8x7B: 11.0 (#301)

BenchmarkGranite 3.1 8b InstructMixtral 8x7B
LMArena Expert11421088
GPQA Diamond—30.6%
MMLU-Pro—33.5%
GPQA (HELM)—29.6%
ARC (AI2) Challenge—87.3%
MMLU—70.6%
OpenBookQA—85.8%
TriviaQA—82.2%

Multilingual Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 30.9 (#260), Mixtral 8x7B: 29.6 (#266)

BenchmarkGranite 3.1 8b InstructMixtral 8x7B
LMArena Non-English10991077
LMArena Chinese11451055
LMArena Russian10921090
LMArena French—1166
LMArena German—1114
LMArena Japanese—931
LMArena Korean—968
LMArena Spanish—1111

Instruction Following Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 58.6 (#259), Mixtral 8x7B: 51.0 (#297)

BenchmarkGranite 3.1 8b InstructMixtral 8x7B
LMArena Instruction Following11311109
IFEval—57.5%

Long Context Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 35.2 (#241), Mixtral 8x7B: 33.4 (#260)

BenchmarkGranite 3.1 8b InstructMixtral 8x7B
LMArena Longer Query11621103

Writing & Preference Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 35.5 (#266), Mixtral 8x7B: 34.2 (#270)

BenchmarkGranite 3.1 8b InstructMixtral 8x7B
LMArena Text11501132
LMArena Creative Writing11291109
LMArena Multi-Turn11081115
WildBench—67.3%

Frequently asked questions

Is Granite 3.1 8b Instruct better than Mixtral 8x7B?

Granite 3.1 8b Instruct is the stronger model overall, scoring 32.4 to 27.1 on the Noometry Index.

Is Granite 3.1 8b Instruct or Mixtral 8x7B better for coding?

Granite 3.1 8b Instruct scores higher on coding benchmarks: 34.5 versus 32.8 in the Noometry coding category.

How many benchmarks do Granite 3.1 8b Instruct and Mixtral 8x7B share?

12 benchmarks have published results for both models. Granite 3.1 8b Instruct has 13 scored results on Noometry and Mixtral 8x7B has 38.

Related comparisons

Go deeper