Noometry

Model comparison

Granite 3.1 8b Instruct vs Llama 2-13B

Granite 3.1 8b Instruct is the stronger model overall, scoring 32.4 to 29.6 on the Noometry Index.

Last verified . 12 shared benchmarks.

Granite 3.1 8b Instruct IBM

32.4

Rank #258 Confirmed

Llama 2-13B Meta

29.6

Rank #309 Confirmed

Summary

  • They share 12 benchmarks with published results for both. Granite 3.1 8b Instruct scores higher in 8 categories and Llama 2-13B in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in reasoning, where Granite 3.1 8b Instruct leads 22.1 to 12.8.

Side by side

Granite 3.1 8b InstructLlama 2-13B
ProviderIBMMeta
Noometry Index32.429.6
Released—2023-07-18
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked1332

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 34.5 (#233), Llama 2-13B: 30.9 (#291)

BenchmarkGranite 3.1 8b InstructLlama 2-13B
LMArena Coding11861062

Agentic & Tool Use Not comparable

Granite 3.1 8b Instruct: 24.1 (#120), Llama 2-13B: —

BenchmarkGranite 3.1 8b InstructLlama 2-13B
Berkeley Function Calling Leaderboard27.1%—

Reasoning Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 22.1 (#207), Llama 2-13B: 12.8 (#337)

BenchmarkGranite 3.1 8b InstructLlama 2-13B
LMArena Hard Prompts11451051
Chess Puzzles—0%
DTBench—42.2%
BIG-Bench Hard—58.2%
Epoch Capabilities Index—106.17
HellaSwag—80.7%
LAMBADA—76.5%
PIQA—80.8%
WinoGrande—72.8%

Math Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 33.0 (#209), Llama 2-13B: 31.1 (#229)

BenchmarkGranite 3.1 8b InstructLlama 2-13B
LMArena Math11521065
GSM8K—36.9%

Knowledge Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 31.1 (#220), Llama 2-13B: 28.1 (#249)

BenchmarkGranite 3.1 8b InstructLlama 2-13B
LMArena Expert11421030
ARC (AI2) Challenge—60.3%
BoolQ—82.4%
MMLU—55.6%
OpenBookQA—57%
TriviaQA—79.6%

Multimodal Not comparable

Granite 3.1 8b Instruct: —, Llama 2-13B: —

BenchmarkGranite 3.1 8b InstructLlama 2-13B
ScienceQA—55.8%

Multilingual Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 30.9 (#260), Llama 2-13B: 26.5 (#279)

BenchmarkGranite 3.1 8b InstructLlama 2-13B
LMArena Non-English10991024
LMArena Chinese11451001
LMArena Russian10921055
LMArena French—1044
LMArena German—1009
LMArena Japanese—894
LMArena Korean—953
LMArena Spanish—1087

Instruction Following Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 58.6 (#259), Llama 2-13B: 53.3 (#287)

BenchmarkGranite 3.1 8b InstructLlama 2-13B
LMArena Instruction Following11311045

Long Context Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 35.2 (#241), Llama 2-13B: 32.3 (#269)

BenchmarkGranite 3.1 8b InstructLlama 2-13B
LMArena Longer Query11621064

Writing & Preference Granite 3.1 8b Instruct leads

Granite 3.1 8b Instruct: 35.5 (#266), Llama 2-13B: 29.8 (#289)

BenchmarkGranite 3.1 8b InstructLlama 2-13B
LMArena Text11501084
LMArena Creative Writing11291047
LMArena Multi-Turn11081050

Frequently asked questions

Is Granite 3.1 8b Instruct better than Llama 2-13B?

Granite 3.1 8b Instruct is the stronger model overall, scoring 32.4 to 29.6 on the Noometry Index.

Is Granite 3.1 8b Instruct or Llama 2-13B better for coding?

Granite 3.1 8b Instruct scores higher on coding benchmarks: 34.5 versus 30.9 in the Noometry coding category.

How many benchmarks do Granite 3.1 8b Instruct and Llama 2-13B share?

12 benchmarks have published results for both models. Granite 3.1 8b Instruct has 13 scored results on Noometry and Llama 2-13B has 32.

Related comparisons

Go deeper