Noometry

Model comparison

DeepSeek-V2.5 (Sep 2024) vs Llama 3-70B

DeepSeek-V2.5 (Sep 2024) is the stronger model overall, scoring 37.6 to 28.8 on the Noometry Index.

Last verified . 21 shared benchmarks.

DeepSeek-V2.5 (Sep 2024) DeepSeek

37.6

Rank #200 Confirmed

Llama 3-70B Meta

28.8

Rank #323 Confirmed

Summary

  • They share 21 benchmarks with published results for both. DeepSeek-V2.5 (Sep 2024) scores higher in 7 categories and Llama 3-70B in 1 category; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where DeepSeek-V2.5 (Sep 2024) leads 35.9 to 12.8.
  • The biggest single-benchmark swing is BigCodeBench Instruct: 48.6% for DeepSeek-V2.5 (Sep 2024) and 43.6% for Llama 3-70B.

Side by side

DeepSeek-V2.5 (Sep 2024)Llama 3-70B
ProviderDeepSeekMeta
Noometry Index37.628.8
Released2024-09-062024-04-18
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked2231

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Llama 3-70B leads

DeepSeek-V2.5 (Sep 2024): 31.7 (#281), Llama 3-70B: 35.8 (#218)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3-70B
BigCodeBench Instruct48.6%43.6%
LMArena Coding13091206
BigCodeBench Complete53.2%54.5%
HumanEval+83.5%72%
MBPP+74.1%69%
Aider Polyglot17.8%—

Agentic & Tool Use Not comparable

DeepSeek-V2.5 (Sep 2024): —, Llama 3-70B: 21.1 (#139)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3-70B
Cybench—5%

Reasoning DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 25.6 (#145), Llama 3-70B: 18.0 (#288)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3-70B
LMArena Hard Prompts12891195
Kagi LLM Benchmark—35.1%
DTBench—54.2%
Epoch Capabilities Index—122.93
ForecastBench—57.1
WinoGrande—83.5%

Math DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 35.9 (#177), Llama 3-70B: 12.8 (#305)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3-70B
LMArena Math12881218
OTIS Mock AIME 2024-2025—4.3%
MATH Level 5—22.6%

Knowledge DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 34.8 (#193), Llama 3-70B: 20.8 (#277)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3-70B
LMArena Expert12661149
GPQA Diamond—40.6%
MMLU—79.3%

Multilingual DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 42.5 (#193), Llama 3-70B: 33.6 (#251)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3-70B
LMArena Non-English12731142
LMArena Chinese13181114
LMArena French12891232
LMArena German12581169
LMArena Japanese12281017
LMArena Korean12091017
LMArena Russian12891159
LMArena Spanish12481241

Instruction Following DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 67.5 (#194), Llama 3-70B: 62.5 (#238)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3-70B
LMArena Instruction Following12801194

Long Context DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 39.5 (#174), Llama 3-70B: 35.6 (#240)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3-70B
LMArena Longer Query13011174

Writing & Preference DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 49.8 (#187), Llama 3-70B: 42.8 (#231)

BenchmarkDeepSeek-V2.5 (Sep 2024)Llama 3-70B
LMArena Text12941221
LMArena Creative Writing12851210
LMArena Multi-Turn12971223

Frequently asked questions

Is DeepSeek-V2.5 (Sep 2024) better than Llama 3-70B?

DeepSeek-V2.5 (Sep 2024) is the stronger model overall, scoring 37.6 to 28.8 on the Noometry Index.

Is DeepSeek-V2.5 (Sep 2024) or Llama 3-70B better for coding?

Llama 3-70B scores higher on coding benchmarks: 35.8 versus 31.7 in the Noometry coding category.

How many benchmarks do DeepSeek-V2.5 (Sep 2024) and Llama 3-70B share?

21 benchmarks have published results for both models. DeepSeek-V2.5 (Sep 2024) has 22 scored results on Noometry and Llama 3-70B has 31.

Related comparisons

Go deeper