Noometry

Model comparison

Granite 4.0 H Small vs Llama 3.1-8B

Granite 4.0 H Small is the stronger model overall, scoring 36.5 to 23.0 on the Noometry Index.

Last verified . 18 shared benchmarks.

Granite 4.0 H Small IBM

36.5

Rank #214 Confirmed

Llama 3.1-8B Meta

23.0

Rank #352 Confirmed

Summary

  • They share 18 benchmarks with published results for both. Granite 4.0 H Small scores higher in 8 categories and Llama 3.1-8B in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Granite 4.0 H Small leads 32.0 to 8.0.
  • The biggest single-benchmark swing is MMLU-Pro: 56.9% for Granite 4.0 H Small and 40.6% for Llama 3.1-8B.

Side by side

Granite 4.0 H SmallLlama 3.1-8B
ProviderIBMMeta
Noometry Index36.523.0
Released—2024-07-23
WeightsOpenOpen
Context window—128K
Max output—4K
Input $ / M tokens—$0.05
Output $ / M tokens—$0.08
Results tracked1943

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Granite 4.0 H Small leads

Granite 4.0 H Small: 36.4 (#209), Llama 3.1-8B: 20.2 (#340)

BenchmarkGranite 4.0 H SmallLlama 3.1-8B
LMArena Coding12491195
SciCode—13.2%
WeirdML—1.7%
BigCodeBench Instruct—32.8%
BigCodeBench Complete—40.5%
HumanEval+—62.8%
MBPP+—55.6%

Agentic & Tool Use Not comparable

Granite 4.0 H Small: —, Llama 3.1-8B: 22.5 (#131)

BenchmarkGranite 4.0 H SmallLlama 3.1-8B
Berkeley Function Calling Leaderboard—25.8%
BALROG—15.1%

Reasoning Granite 4.0 H Small leads

Granite 4.0 H Small: 24.4 (#163), Llama 3.1-8B: 14.9 (#321)

BenchmarkGranite 4.0 H SmallLlama 3.1-8B
LMArena Hard Prompts12401175
CritPt—0%
Chess Puzzles—0%
DTBench—50.9%
LMCA—5.4%
Epoch Capabilities Index—116.57
PIQA—81.2%

Math Granite 4.0 H Small leads

Granite 4.0 H Small: 31.4 (#223), Llama 3.1-8B: 10.2 (#317)

BenchmarkGranite 4.0 H SmallLlama 3.1-8B
Omni-MATH29.6%13.7%
LMArena Math12471179
OTIS Mock AIME 2024-2025—1.7%
MATH Level 5—22.9%
GSM8K—82.4%

Knowledge Granite 4.0 H Small leads

Granite 4.0 H Small: 32.0 (#215), Llama 3.1-8B: 8.0 (#307)

BenchmarkGranite 4.0 H SmallLlama 3.1-8B
MMLU-Pro56.9%40.6%
GPQA (HELM)38.3%24.7%
LMArena Expert12511144
GPQA Diamond—27%
Vectara Hallucination Rate5.2%—
BoolQ—82.8%
MMLU—56.1%

Multilingual Granite 4.0 H Small leads

Granite 4.0 H Small: 38.6 (#228), Llama 3.1-8B: 34.0 (#249)

BenchmarkGranite 4.0 H SmallLlama 3.1-8B
LMArena Non-English12161148
LMArena Chinese12491151
LMArena Russian12011158
LMArena Spanish12581169
LMArena French—1177
LMArena German—1144
LMArena Japanese—1061
LMArena Korean—1053

Instruction Following Granite 4.0 H Small leads

Granite 4.0 H Small: 68.7 (#183), Llama 3.1-8B: 58.9 (#258)

BenchmarkGranite 4.0 H SmallLlama 3.1-8B
IFEval89%74.3%
LMArena Instruction Following12221159

Long Context Granite 4.0 H Small leads

Granite 4.0 H Small: 37.7 (#213), Llama 3.1-8B: 35.8 (#238)

BenchmarkGranite 4.0 H SmallLlama 3.1-8B
LMArena Longer Query12421182

Writing & Preference Granite 4.0 H Small leads

Granite 4.0 H Small: 43.8 (#227), Llama 3.1-8B: 29.7 (#290)

BenchmarkGranite 4.0 H SmallLlama 3.1-8B
LMArena Text12411187
LMArena Creative Writing12111154
WildBench73.9%68.7%
LMArena Multi-Turn12421172
EQ-Bench Creative Writing—713

Frequently asked questions

Is Granite 4.0 H Small better than Llama 3.1-8B?

Granite 4.0 H Small is the stronger model overall, scoring 36.5 to 23.0 on the Noometry Index.

Is Granite 4.0 H Small or Llama 3.1-8B better for coding?

Granite 4.0 H Small scores higher on coding benchmarks: 36.4 versus 20.2 in the Noometry coding category.

How many benchmarks do Granite 4.0 H Small and Llama 3.1-8B share?

18 benchmarks have published results for both models. Granite 4.0 H Small has 19 scored results on Noometry and Llama 3.1-8B has 43.

Related comparisons

Go deeper