Noometry

Model comparison

DeepSeek-V2.5 (Sep 2024) vs Llama 3.1-70B

DeepSeek-V2.5 (Sep 2024) is the stronger model overall, scoring 37.0 to 29.4 on the Noometry Index.

Last verified . 19 shared benchmarks.

DeepSeek-V2.5 (Sep 2024) DeepSeek

37.0

Rank #197 Confirmed

Llama 3.1-70B Meta

29.4

Rank #299 Confirmed

Summary

  • They share 19 benchmarks with published results for both. DeepSeek-V2.5 (Sep 2024) scores higher in 8 categories and Llama 3.1-70B in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where DeepSeek-V2.5 (Sep 2024) leads 33.3 to 12.7.

Side by side

DeepSeek-V2.5 (Sep 2024)Llama 3.1-70B
ProviderDeepSeekMeta
Noometry Index37.029.4
Released2024-09-062024-07-23
WeightsOpenOpen
Context window—128K
Max output—4K
Input $ / M tokens—$0.40
Output $ / M tokens—$0.40
Results tracked2235

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 31.3 (#274), Llama 3.1-70B: 29.8 (#292)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3.1-70B
BigCodeBench Instruct48.6%46.1%
LMArena Coding13101260
BigCodeBench Complete53.2%54.8%
Aider Polyglot17.8%—
WeirdML—9%
HumanEval+83.5%—
MBPP+74.1%—

Agentic & Tool Use Not comparable

DeepSeek-V2.5 (Sep 2024): —, Llama 3.1-70B: 24.9 (#104)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3.1-70B
TheAgentCompany—6.9%
BALROG—27.9%

Reasoning DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 24.5 (#145), Llama 3.1-70B: 20.8 (#215)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3.1-70B
LMArena Hard Prompts12891241
DTBench—60%
LMCA—14.8%
Epoch Capabilities Index—125.91

Math DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 33.3 (#177), Llama 3.1-70B: 12.7 (#298)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3.1-70B
LMArena Math12891252
OTIS Mock AIME 2024-2025—3.6%
Omni-MATH—21%
MATH Level 5—36.7%

Knowledge DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 34.2 (#190), Llama 3.1-70B: 24.1 (#264)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3.1-70B
LMArena Expert12661209
GPQA Diamond—44.2%
MMLU-Pro—65.3%
GPQA (HELM)—42.6%
MMLU—80.1%

Multilingual DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 42.7 (#188), Llama 3.1-70B: 38.9 (#220)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3.1-70B
LMArena Non-English12731219
LMArena Chinese13171215
LMArena French12881260
LMArena German12581222
LMArena Japanese12281132
LMArena Korean12091140
LMArena Russian12891234
LMArena Spanish12481253

Instruction Following DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 67.7 (#189), Llama 3.1-70B: 65.5 (#218)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3.1-70B
LMArena Instruction Following12801231
IFEval—82.1%

Long Context DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 39.5 (#169), Llama 3.1-70B: 37.6 (#208)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3.1-70B
LMArena Longer Query13011241

Writing & Preference DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 50.0 (#182), Llama 3.1-70B: 35.6 (#262)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3.1-70B
LMArena Text12941261
LMArena Creative Writing12851232
LMArena Multi-Turn12971256
EQ-Bench Creative Writing—784
WildBench—75.8%

Frequently asked questions

Is DeepSeek-V2.5 (Sep 2024) better than Llama 3.1-70B?

DeepSeek-V2.5 (Sep 2024) is the stronger model overall, scoring 37.0 to 29.4 on the Noometry Index.

Is DeepSeek-V2.5 (Sep 2024) or Llama 3.1-70B better for coding?

DeepSeek-V2.5 (Sep 2024) scores higher on coding benchmarks: 31.3 versus 29.8 in the Noometry coding category.

How many benchmarks do DeepSeek-V2.5 (Sep 2024) and Llama 3.1-70B share?

19 benchmarks have published results for both models. DeepSeek-V2.5 (Sep 2024) has 22 scored results on Noometry and Llama 3.1-70B has 35.

Related comparisons

Go deeper