Noometry

Model comparison

Deepseek Coder v2 vs Llama 3.1-8B

Deepseek Coder v2 is the stronger model overall, scoring 35.9 to 23.0 on the Noometry Index.

Last verified . 22 shared benchmarks.

Deepseek Coder v2 DeepSeek

35.9

Rank #220 Confirmed

Llama 3.1-8B Meta

23.0

Rank #352 Confirmed

Summary

  • They share 22 benchmarks with published results for both. Deepseek Coder v2 scores higher in 8 categories and Llama 3.1-8B in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where Deepseek Coder v2 leads 34.9 to 10.2.
  • The biggest single-benchmark swing is BigCodeBench Complete: 59.7% for Deepseek Coder v2 and 40.5% for Llama 3.1-8B.

Side by side

Deepseek Coder v2Llama 3.1-8B
ProviderDeepSeekMeta
Noometry Index35.923.0
Released2024-06-172024-07-23
WeightsOpenOpen
Context window—128K
Max output—4K
Input $ / M tokens—$0.05
Output $ / M tokens—$0.08
Results tracked2443

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Deepseek Coder v2 leads

Deepseek Coder v2: 38.1 (#183), Llama 3.1-8B: 20.2 (#340)

BenchmarkDeepseek Coder v2Llama 3.1-8B
BigCodeBench Instruct48.2%32.8%
LMArena Coding12511195
BigCodeBench Complete59.7%40.5%
HumanEval+82.3%62.8%
MBPP+75.1%55.6%
SciCode—13.2%
WeirdML—1.7%

Agentic & Tool Use Not comparable

Deepseek Coder v2: —, Llama 3.1-8B: 22.5 (#131)

BenchmarkDeepseek Coder v2Llama 3.1-8B
Berkeley Function Calling Leaderboard—25.8%
BALROG—15.1%

Reasoning Deepseek Coder v2 leads

Deepseek Coder v2: 23.6 (#176), Llama 3.1-8B: 14.9 (#321)

BenchmarkDeepseek Coder v2Llama 3.1-8B
LMArena Hard Prompts12071175
CritPt—0%
Chess Puzzles—0%
DTBench—50.9%
LMCA—5.4%
Epoch Capabilities Index—116.57
PIQA—81.2%
WinoGrande83.7%—

Math Deepseek Coder v2 leads

Deepseek Coder v2: 34.9 (#190), Llama 3.1-8B: 10.2 (#317)

BenchmarkDeepseek Coder v2Llama 3.1-8B
LMArena Math12411179
GSM8K94.5%82.4%
OTIS Mock AIME 2024-2025—1.7%
Omni-MATH—13.7%
MATH Level 5—22.9%

Knowledge Deepseek Coder v2 leads

Deepseek Coder v2: 32.3 (#212), Llama 3.1-8B: 8.0 (#307)

BenchmarkDeepseek Coder v2Llama 3.1-8B
LMArena Expert11811144
GPQA Diamond—27%
MMLU-Pro—40.6%
GPQA (HELM)—24.7%
ARC (AI2) Challenge64.3%—
BoolQ—82.8%
MMLU—56.1%

Multilingual Deepseek Coder v2 leads

Deepseek Coder v2: 36.3 (#240), Llama 3.1-8B: 34.0 (#249)

BenchmarkDeepseek Coder v2Llama 3.1-8B
LMArena Non-English11821148
LMArena Chinese12011151
LMArena French11851177
LMArena German11641144
LMArena Japanese11261061
LMArena Korean11041053
LMArena Russian11881158
LMArena Spanish11531169

Instruction Following Deepseek Coder v2 leads

Deepseek Coder v2: 61.7 (#242), Llama 3.1-8B: 58.9 (#258)

BenchmarkDeepseek Coder v2Llama 3.1-8B
LMArena Instruction Following11801159
IFEval—74.3%

Long Context Deepseek Coder v2 leads

Deepseek Coder v2: 37.0 (#224), Llama 3.1-8B: 35.8 (#238)

BenchmarkDeepseek Coder v2Llama 3.1-8B
LMArena Longer Query12191182

Writing & Preference Deepseek Coder v2 leads

Deepseek Coder v2: 38.2 (#253), Llama 3.1-8B: 29.7 (#290)

BenchmarkDeepseek Coder v2Llama 3.1-8B
LMArena Text11911187
LMArena Creative Writing11201154
LMArena Multi-Turn11771172
EQ-Bench Creative Writing—713
WildBench—68.7%

Frequently asked questions

Is Deepseek Coder v2 better than Llama 3.1-8B?

Deepseek Coder v2 is the stronger model overall, scoring 35.9 to 23.0 on the Noometry Index.

Is Deepseek Coder v2 or Llama 3.1-8B better for coding?

Deepseek Coder v2 scores higher on coding benchmarks: 38.1 versus 20.2 in the Noometry coding category.

How many benchmarks do Deepseek Coder v2 and Llama 3.1-8B share?

22 benchmarks have published results for both models. Deepseek Coder v2 has 24 scored results on Noometry and Llama 3.1-8B has 43.

Related comparisons

Go deeper