Noometry

Model comparison

DeepSeek-V2.5 (Sep 2024) vs Granite 3.1 8b Instruct

DeepSeek-V2.5 (Sep 2024) is the stronger model overall, scoring 37.0 to 31.8 on the Noometry Index.

Last verified . 12 shared benchmarks.

DeepSeek-V2.5 (Sep 2024) DeepSeek

37.0

Rank #197 Confirmed

Granite 3.1 8b Instruct IBM

31.8

Rank #256 Confirmed

Summary

  • They share 12 benchmarks with published results for both. DeepSeek-V2.5 (Sep 2024) scores higher in 7 categories and Granite 3.1 8b Instruct in 1 category; 8 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where DeepSeek-V2.5 (Sep 2024) leads 50.0 to 35.7.

Side by side

DeepSeek-V2.5 (Sep 2024)Granite 3.1 8b Instruct
ProviderDeepSeekIBM
Noometry Index37.031.8
Released2024-09-06—
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked2213

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Granite 3.1 8b Instruct leads

DeepSeek-V2.5 (Sep 2024): 31.3 (#274), Granite 3.1 8b Instruct: 33.9 (#232)

BenchmarkDeepSeek-V2.5 (Sep 2024)Granite 3.1 8b Instruct
LMArena Coding13101186
Aider Polyglot17.8%—
BigCodeBench Instruct48.6%—
BigCodeBench Complete53.2%—
HumanEval+83.5%—
MBPP+74.1%—

Agentic & Tool Use Not comparable

DeepSeek-V2.5 (Sep 2024): —, Granite 3.1 8b Instruct: 23.9 (#113)

BenchmarkDeepSeek-V2.5 (Sep 2024)Granite 3.1 8b Instruct
Berkeley Function Calling Leaderboard—27.1%

Reasoning DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 24.5 (#145), Granite 3.1 8b Instruct: 21.2 (#204)

BenchmarkDeepSeek-V2.5 (Sep 2024)Granite 3.1 8b Instruct
LMArena Hard Prompts12891146

Math DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 33.3 (#177), Granite 3.1 8b Instruct: 30.4 (#210)

BenchmarkDeepSeek-V2.5 (Sep 2024)Granite 3.1 8b Instruct
LMArena Math12891152

Knowledge DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 34.2 (#190), Granite 3.1 8b Instruct: 30.6 (#218)

BenchmarkDeepSeek-V2.5 (Sep 2024)Granite 3.1 8b Instruct
LMArena Expert12661142

Multilingual DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 42.7 (#188), Granite 3.1 8b Instruct: 31.1 (#255)

BenchmarkDeepSeek-V2.5 (Sep 2024)Granite 3.1 8b Instruct
LMArena Non-English12731099
LMArena Chinese13171145
LMArena Russian12891092
LMArena French1288—
LMArena German1258—
LMArena Japanese1228—
LMArena Korean1209—
LMArena Spanish1248—

Instruction Following DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 67.7 (#189), Granite 3.1 8b Instruct: 58.9 (#254)

BenchmarkDeepSeek-V2.5 (Sep 2024)Granite 3.1 8b Instruct
LMArena Instruction Following12801131

Long Context DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 39.5 (#169), Granite 3.1 8b Instruct: 35.2 (#236)

BenchmarkDeepSeek-V2.5 (Sep 2024)Granite 3.1 8b Instruct
LMArena Longer Query13011162

Writing & Preference DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 50.0 (#182), Granite 3.1 8b Instruct: 35.7 (#261)

BenchmarkDeepSeek-V2.5 (Sep 2024)Granite 3.1 8b Instruct
LMArena Text12941150
LMArena Creative Writing12851129
LMArena Multi-Turn12971108

Frequently asked questions

Is DeepSeek-V2.5 (Sep 2024) better than Granite 3.1 8b Instruct?

DeepSeek-V2.5 (Sep 2024) is the stronger model overall, scoring 37.0 to 31.8 on the Noometry Index.

Is DeepSeek-V2.5 (Sep 2024) or Granite 3.1 8b Instruct better for coding?

Granite 3.1 8b Instruct scores higher on coding benchmarks: 33.9 versus 31.3 in the Noometry coding category.

How many benchmarks do DeepSeek-V2.5 (Sep 2024) and Granite 3.1 8b Instruct share?

12 benchmarks have published results for both models. DeepSeek-V2.5 (Sep 2024) has 22 scored results on Noometry and Granite 3.1 8b Instruct has 13.

Related comparisons

Go deeper