Noometry

Model comparison

DeepSeek LLM 67B vs Llama 3.1-405B

Llama 3.1-405B is the stronger model overall, scoring 30.7 to 24.9 on the Noometry Index.

Last verified . 14 shared benchmarks.

DeepSeek LLM 67B DeepSeek

24.9

Rank #347 Confirmed

Llama 3.1-405B Meta

30.7

Rank #288 Confirmed

Summary

  • They share 14 benchmarks with published results for both. DeepSeek LLM 67B scores higher in 0 categories and Llama 3.1-405B in 8 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Llama 3.1-405B leads 30.4 to 7.0.
  • The biggest single-benchmark swing is MATH Level 5: 6.4% for DeepSeek LLM 67B and 49.8% for Llama 3.1-405B.

Side by side

DeepSeek LLM 67BLlama 3.1-405B
ProviderDeepSeekMeta
Noometry Index24.930.7
Released2023-11-292024-07-23
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked1542

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Llama 3.1-405B leads

DeepSeek LLM 67B: 31.9 (#278), Llama 3.1-405B: 33.1 (#262)

BenchmarkDeepSeek LLM 67BLlama 3.1-405B
LMArena Coding10961291
WeirdML—21.4%

Agentic & Tool Use Not comparable

DeepSeek LLM 67B: —, Llama 3.1-405B: 21.0 (#140)

BenchmarkDeepSeek LLM 67BLlama 3.1-405B
TheAgentCompany—7.4%
Cybench—7.5%

Reasoning Too close to call

DeepSeek LLM 67B: 16.5 (#304), Llama 3.1-405B: 16.8 (#300)

BenchmarkDeepSeek LLM 67BLlama 3.1-405B
LMArena Hard Prompts10701269
Epoch Capabilities Index110.5128.75
SimpleBench—23%
Kagi LLM Benchmark—45%
Chess Puzzles0%—
DTBench—61.4%
BIG-Bench Hard—82.9%
ForecastBench—59.9
HellaSwag—89.2%
PIQA—85.9%
WinoGrande—89.2%

Math Llama 3.1-405B leads

DeepSeek LLM 67B: 8.7 (#324), Llama 3.1-405B: 18.4 (#290)

BenchmarkDeepSeek LLM 67BLlama 3.1-405B
OTIS Mock AIME 2024-20250.8%9.7%
LMArena Math11081281
MATH Level 56.4%49.8%
Omni-MATH—24.9%

Knowledge Llama 3.1-405B leads

DeepSeek LLM 67B: 7.0 (#313), Llama 3.1-405B: 30.4 (#227)

BenchmarkDeepSeek LLM 67BLlama 3.1-405B
GPQA Diamond24.6%50.9%
MMLU-Pro—72.3%
Confabulations—17.6%
GPQA (HELM)—52.2%
LMArena Expert—1243
ARC (AI2) Challenge—95.3%
MMLU—84.5%
TriviaQA—82.7%

Multilingual Llama 3.1-405B leads

DeepSeek LLM 67B: 29.4 (#267), Llama 3.1-405B: 40.7 (#214)

BenchmarkDeepSeek LLM 67BLlama 3.1-405B
LMArena Non-English10731248
LMArena Chinese11321242
LMArena French—1279
LMArena German—1252
LMArena Japanese—1208
LMArena Korean—1184
LMArena Russian—1265
LMArena Spanish—1260

Instruction Following Llama 3.1-405B leads

DeepSeek LLM 67B: 55.4 (#277), Llama 3.1-405B: 65.9 (#214)

BenchmarkDeepSeek LLM 67BLlama 3.1-405B
LMArena Instruction Following10791259
IFEval—81.1%

Long Context Llama 3.1-405B leads

DeepSeek LLM 67B: 33.1 (#265), Llama 3.1-405B: 38.4 (#197)

BenchmarkDeepSeek LLM 67BLlama 3.1-405B
LMArena Longer Query10921266

Writing & Preference Llama 3.1-405B leads

DeepSeek LLM 67B: 31.6 (#282), Llama 3.1-405B: 38.9 (#251)

BenchmarkDeepSeek LLM 67BLlama 3.1-405B
LMArena Text11051284
LMArena Creative Writing10671262
LMArena Multi-Turn10821297
EQ-Bench Creative Writing—870
WildBench—78.3%

Frequently asked questions

Is DeepSeek LLM 67B better than Llama 3.1-405B?

Llama 3.1-405B is the stronger model overall, scoring 30.7 to 24.9 on the Noometry Index.

Is DeepSeek LLM 67B or Llama 3.1-405B better for coding?

Llama 3.1-405B scores higher on coding benchmarks: 33.1 versus 31.9 in the Noometry coding category.

How many benchmarks do DeepSeek LLM 67B and Llama 3.1-405B share?

14 benchmarks have published results for both models. DeepSeek LLM 67B has 15 scored results on Noometry and Llama 3.1-405B has 42.

Related comparisons

Go deeper