Noometry

Model comparison

Granite 3.0 8b Instruct vs Mistral Nemo

Granite 3.0 8b Instruct is the stronger model overall, scoring 31.6 to 26.4 on the Noometry Index.

Last verified . 0 shared benchmarks.

Granite 3.0 8b Instruct IBM

31.6

Rank #270 Confirmed

Mistral Nemo Mistral AI

26.4

Rank #337 Confirmed

Summary

  • The widest gap is in knowledge, where Granite 3.0 8b Instruct leads 29.6 to 12.3.

Side by side

Granite 3.0 8b InstructMistral Nemo
ProviderIBMMistral AI
Noometry Index31.626.4
Released—2024-07-01
WeightsOpenOpen
Context window—128K
Max output—128K
Input $ / M tokens—$0.15
Output $ / M tokens—$0.15
Results tracked1410

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Granite 3.0 8b Instruct: 29.7 (#301), Mistral Nemo: —

BenchmarkGranite 3.0 8b InstructMistral Nemo
BigCodeBench Instruct29.3%—
LMArena Coding1112—
BigCodeBench Complete35.4%—

Agentic & Tool Use Not comparable

Granite 3.0 8b Instruct: —, Mistral Nemo: 23.5 (#125)

BenchmarkGranite 3.0 8b InstructMistral Nemo
Berkeley Function Calling Leaderboard—27.6%
BALROG—17.6%

Reasoning Too close to call

Granite 3.0 8b Instruct: 20.9 (#230), Mistral Nemo: 20.7 (#232)

BenchmarkGranite 3.0 8b InstructMistral Nemo
LMArena Hard Prompts1092—
DTBench—48.6%
Epoch Capabilities Index—118.68
PIQA—83.5%

Math Granite 3.0 8b Instruct leads

Granite 3.0 8b Instruct: 32.8 (#210), Mistral Nemo: 25.5 (#268)

BenchmarkGranite 3.0 8b InstructMistral Nemo
LMArena Math1143—
MATH Level 5—10.8%
GSM8K—84.2%

Knowledge Granite 3.0 8b Instruct leads

Granite 3.0 8b Instruct: 29.6 (#236), Mistral Nemo: 12.3 (#298)

BenchmarkGranite 3.0 8b InstructMistral Nemo
GPQA Diamond—29.9%
LMArena Expert1087—
BoolQ—82.5%

Multilingual Not comparable

Granite 3.0 8b Instruct: 27.2 (#276), Mistral Nemo: —

BenchmarkGranite 3.0 8b InstructMistral Nemo
LMArena Non-English1037—
LMArena Chinese1063—
LMArena Russian1060—

Instruction Following Not comparable

Granite 3.0 8b Instruct: 56.0 (#276), Mistral Nemo: —

BenchmarkGranite 3.0 8b InstructMistral Nemo
LMArena Instruction Following1088—

Long Context Not comparable

Granite 3.0 8b Instruct: 34.0 (#252), Mistral Nemo: —

BenchmarkGranite 3.0 8b InstructMistral Nemo
LMArena Longer Query1122—

Writing & Preference Granite 3.0 8b Instruct leads

Granite 3.0 8b Instruct: 31.1 (#285), Mistral Nemo: 28.5 (#296)

BenchmarkGranite 3.0 8b InstructMistral Nemo
LMArena Text1096—
LMArena Creative Writing1071—
EQ-Bench Creative Writing—881
LMArena Multi-Turn1063—

Frequently asked questions

Is Granite 3.0 8b Instruct better than Mistral Nemo?

Granite 3.0 8b Instruct is the stronger model overall, scoring 31.6 to 26.4 on the Noometry Index.

How many benchmarks do Granite 3.0 8b Instruct and Mistral Nemo share?

0 benchmarks have published results for both models. Granite 3.0 8b Instruct has 14 scored results on Noometry and Mistral Nemo has 10.

Related comparisons

Go deeper