Noometry

Model comparison

DeepSeek LLM 67B vs Llama 3.1-8B

DeepSeek LLM 67B is the stronger model overall, scoring 24.9 to 23.0 on the Noometry Index.

Last verified . 15 shared benchmarks.

DeepSeek LLM 67B DeepSeek

24.9

Rank #347 Confirmed

Llama 3.1-8B Meta

23.0

Rank #352 Confirmed

Summary

  • They share 15 benchmarks with published results for both. DeepSeek LLM 67B scores higher in 3 categories and Llama 3.1-8B in 5 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in coding, where DeepSeek LLM 67B leads 31.9 to 20.2.
  • The biggest single-benchmark swing is MATH Level 5: 6.4% for DeepSeek LLM 67B and 22.9% for Llama 3.1-8B.

Side by side

DeepSeek LLM 67BLlama 3.1-8B
ProviderDeepSeekMeta
Noometry Index24.923.0
Released2023-11-292024-07-23
WeightsOpenOpen
Context window—128K
Max output—4K
Input $ / M tokens—$0.05
Output $ / M tokens—$0.08
Results tracked1543

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding DeepSeek LLM 67B leads

DeepSeek LLM 67B: 31.9 (#278), Llama 3.1-8B: 20.2 (#340)

BenchmarkDeepSeek LLM 67BLlama 3.1-8B
LMArena Coding10961195
SciCode—13.2%
WeirdML—1.7%
BigCodeBench Instruct—32.8%
BigCodeBench Complete—40.5%
HumanEval+—62.8%
MBPP+—55.6%

Agentic & Tool Use Not comparable

DeepSeek LLM 67B: —, Llama 3.1-8B: 22.5 (#131)

BenchmarkDeepSeek LLM 67BLlama 3.1-8B
Berkeley Function Calling Leaderboard—25.8%
BALROG—15.1%

Reasoning DeepSeek LLM 67B leads

DeepSeek LLM 67B: 16.5 (#304), Llama 3.1-8B: 14.9 (#321)

BenchmarkDeepSeek LLM 67BLlama 3.1-8B
Chess Puzzles0%0%
LMArena Hard Prompts10701175
Epoch Capabilities Index110.5116.57
CritPt—0%
DTBench—50.9%
LMCA—5.4%
PIQA—81.2%

Math Llama 3.1-8B leads

DeepSeek LLM 67B: 8.7 (#324), Llama 3.1-8B: 10.2 (#317)

BenchmarkDeepSeek LLM 67BLlama 3.1-8B
OTIS Mock AIME 2024-20250.8%1.7%
LMArena Math11081179
MATH Level 56.4%22.9%
Omni-MATH—13.7%
GSM8K—82.4%

Knowledge Too close to call

DeepSeek LLM 67B: 7.0 (#313), Llama 3.1-8B: 8.0 (#307)

BenchmarkDeepSeek LLM 67BLlama 3.1-8B
GPQA Diamond24.6%27%
MMLU-Pro—40.6%
GPQA (HELM)—24.7%
LMArena Expert—1144
BoolQ—82.8%
MMLU—56.1%

Multilingual Llama 3.1-8B leads

DeepSeek LLM 67B: 29.4 (#267), Llama 3.1-8B: 34.0 (#249)

BenchmarkDeepSeek LLM 67BLlama 3.1-8B
LMArena Non-English10731148
LMArena Chinese11321151
LMArena French—1177
LMArena German—1144
LMArena Japanese—1061
LMArena Korean—1053
LMArena Russian—1158
LMArena Spanish—1169

Instruction Following Llama 3.1-8B leads

DeepSeek LLM 67B: 55.4 (#277), Llama 3.1-8B: 58.9 (#258)

BenchmarkDeepSeek LLM 67BLlama 3.1-8B
LMArena Instruction Following10791159
IFEval—74.3%

Long Context Llama 3.1-8B leads

DeepSeek LLM 67B: 33.1 (#265), Llama 3.1-8B: 35.8 (#238)

BenchmarkDeepSeek LLM 67BLlama 3.1-8B
LMArena Longer Query10921182

Writing & Preference DeepSeek LLM 67B leads

DeepSeek LLM 67B: 31.6 (#282), Llama 3.1-8B: 29.7 (#290)

BenchmarkDeepSeek LLM 67BLlama 3.1-8B
LMArena Text11051187
LMArena Creative Writing10671154
LMArena Multi-Turn10821172
EQ-Bench Creative Writing—713
WildBench—68.7%

Frequently asked questions

Is DeepSeek LLM 67B better than Llama 3.1-8B?

DeepSeek LLM 67B is the stronger model overall, scoring 24.9 to 23.0 on the Noometry Index.

Is DeepSeek LLM 67B or Llama 3.1-8B better for coding?

DeepSeek LLM 67B scores higher on coding benchmarks: 31.9 versus 20.2 in the Noometry coding category.

How many benchmarks do DeepSeek LLM 67B and Llama 3.1-8B share?

15 benchmarks have published results for both models. DeepSeek LLM 67B has 15 scored results on Noometry and Llama 3.1-8B has 43.

Related comparisons

Go deeper