Noometry

Model comparison

Gemini 1.5 Pro (May 2024) vs Granite 4.1 8b

Granite 4.1 8b is the stronger model overall, scoring 37.4 to 32.1 on the Noometry Index.

Last verified . 12 shared benchmarks.

Gemini 1.5 Pro (May 2024) Google

32.1

Rank #261 Confirmed

Granite 4.1 8b IBM

37.4

Rank #205 Confirmed

Summary

  • They share 12 benchmarks with published results for both. Gemini 1.5 Pro (May 2024) scores higher in 5 categories and Granite 4.1 8b in 3 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in reasoning, where Granite 4.1 8b leads 25.7 to 12.3.
  • Granite 4.1 8b has downloadable open weights; the other is API-only.

Side by side

Gemini 1.5 Pro (May 2024)Granite 4.1 8b
ProviderGoogleIBM
Noometry Index32.137.4
Released2024-02-15—
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked4513

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemini 1.5 Pro (May 2024) leads

Gemini 1.5 Pro (May 2024): 34.2 (#241), Granite 4.1 8b: 30.1 (#297)

BenchmarkGemini 1.5 Pro (May 2024)Granite 4.1 8b
LMArena Coding12941312
LMArena WebDev—1192
WeirdML22.2%—
BigCodeBench Instruct43.8%—
BigCodeBench Complete57.5%—
CadEval34%—
HumanEval+79.3%—
MBPP+74.6%—

Agentic & Tool Use Not comparable

Gemini 1.5 Pro (May 2024): 17.9 (#145), Granite 4.1 8b: —

BenchmarkGemini 1.5 Pro (May 2024)Granite 4.1 8b
TheAgentCompany3.4%—
Cybench7.5%—
BALROG21%—

Reasoning Granite 4.1 8b leads

Gemini 1.5 Pro (May 2024): 12.3 (#338), Granite 4.1 8b: 25.7 (#143)

BenchmarkGemini 1.5 Pro (May 2024)Granite 4.1 8b
LMArena Hard Prompts12961293
ARC-AGI-20.8%—
SimpleBench27.1%—
DTBench59%—
BIG-Bench Hard89.2%—
Epoch Capabilities Index131.73—
ForecastBench58.4—

Math Granite 4.1 8b leads

Gemini 1.5 Pro (May 2024): 25.8 (#266), Granite 4.1 8b: 36.4 (#166)

BenchmarkGemini 1.5 Pro (May 2024)Granite 4.1 8b
LMArena Math13151312
OTIS Mock AIME 2024-202523.1%—
Omni-MATH36.4%—
MATH Level 570.4%—

Knowledge Granite 4.1 8b leads

Gemini 1.5 Pro (May 2024): 29.4 (#239), Granite 4.1 8b: 36.1 (#174)

BenchmarkGemini 1.5 Pro (May 2024)Granite 4.1 8b
LMArena Expert12791309
GPQA Diamond57.2%—
Humanity's Last Exam4.6%—
MMLU-Pro73.7%—
Confabulations13.5%—
GPQA (HELM)53.4%—
MMLU86.9%—

Multimodal Not comparable

Gemini 1.5 Pro (May 2024): 36.8 (#77), Granite 4.1 8b: —

BenchmarkGemini 1.5 Pro (May 2024)Granite 4.1 8b
LMArena Vision1161—
Video-MME75%—

Multilingual Gemini 1.5 Pro (May 2024) leads

Gemini 1.5 Pro (May 2024): 45.3 (#174), Granite 4.1 8b: 41.7 (#204)

BenchmarkGemini 1.5 Pro (May 2024)Granite 4.1 8b
LMArena Non-English13121261
LMArena Chinese13311337
LMArena Russian13201240
LMArena French1302—
LMArena German1286—
LMArena Japanese1292—
LMArena Korean1298—
LMArena Spanish1311—

Instruction Following Gemini 1.5 Pro (May 2024) leads

Gemini 1.5 Pro (May 2024): 68.6 (#185), Granite 4.1 8b: 66.9 (#203)

BenchmarkGemini 1.5 Pro (May 2024)Granite 4.1 8b
LMArena Instruction Following12971269
IFEval83.7%—

Long Context Gemini 1.5 Pro (May 2024) leads

Gemini 1.5 Pro (May 2024): 39.8 (#169), Granite 4.1 8b: 38.7 (#193)

BenchmarkGemini 1.5 Pro (May 2024)Granite 4.1 8b
LMArena Longer Query13081275

Writing & Preference Gemini 1.5 Pro (May 2024) leads

Gemini 1.5 Pro (May 2024): 52.4 (#172), Granite 4.1 8b: 48.1 (#204)

BenchmarkGemini 1.5 Pro (May 2024)Granite 4.1 8b
LMArena Text13191290
LMArena Creative Writing13331251
LMArena Multi-Turn12961268
WildBench81.3%—

Frequently asked questions

Is Gemini 1.5 Pro (May 2024) better than Granite 4.1 8b?

Granite 4.1 8b is the stronger model overall, scoring 37.4 to 32.1 on the Noometry Index.

Is Gemini 1.5 Pro (May 2024) or Granite 4.1 8b better for coding?

Gemini 1.5 Pro (May 2024) scores higher on coding benchmarks: 34.2 versus 30.1 in the Noometry coding category.

How many benchmarks do Gemini 1.5 Pro (May 2024) and Granite 4.1 8b share?

12 benchmarks have published results for both models. Gemini 1.5 Pro (May 2024) has 45 scored results on Noometry and Granite 4.1 8b has 13.

Related comparisons

Go deeper