Noometry

Model comparison

DeepSeek-V2 (MoE-236B, May 2024) vs Llama 3.1 Nemotron 51b Instruct

Llama 3.1 Nemotron 51b Instruct has enough public results to be ranked (#219); DeepSeek-V2 (MoE-236B, May 2024) does not yet, so treat this comparison as directional.

Last verified . 0 shared benchmarks.

DeepSeek-V2 (MoE-236B, May 2024) DeepSeek

39.7

Unranked Sparse

Llama 3.1 Nemotron 51b Instruct NVIDIA

35.2

Rank #219 Confirmed

Summary

  • The widest gap is in coding, where DeepSeek-V2 (MoE-236B, May 2024) leads 39.8 to 35.0.

Side by side

DeepSeek-V2 (MoE-236B, May 2024)Llama 3.1 Nemotron 51b Instruct
ProviderDeepSeekNVIDIA
Noometry Index39.735.2
Released2024-05-07—
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked1012

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding DeepSeek-V2 (MoE-236B, May 2024) leads

DeepSeek-V2 (MoE-236B, May 2024): 39.8 (#141), Llama 3.1 Nemotron 51b Instruct: 35.0 (#219)

BenchmarkDeepSeek-V2 (MoE-236B, May 2024)Llama 3.1 Nemotron 51b Instruct
BigCodeBench Instruct48.9%—
LMArena Coding—1223
BigCodeBench Complete59.4%—

Reasoning Not comparable

DeepSeek-V2 (MoE-236B, May 2024): —, Llama 3.1 Nemotron 51b Instruct: 22.5 (#177)

BenchmarkDeepSeek-V2 (MoE-236B, May 2024)Llama 3.1 Nemotron 51b Instruct
LMArena Hard Prompts—1204
BIG-Bench Hard78.8%—
Epoch Capabilities Index124.78—
HellaSwag87.1%—
PIQA83.9%—
WinoGrande86.3%—

Math Not comparable

DeepSeek-V2 (MoE-236B, May 2024): —, Llama 3.1 Nemotron 51b Instruct: 32.0 (#189)

BenchmarkDeepSeek-V2 (MoE-236B, May 2024)Llama 3.1 Nemotron 51b Instruct
LMArena Math—1230

Knowledge Not comparable

DeepSeek-V2 (MoE-236B, May 2024): —, Llama 3.1 Nemotron 51b Instruct: 31.3 (#213)

BenchmarkDeepSeek-V2 (MoE-236B, May 2024)Llama 3.1 Nemotron 51b Instruct
LMArena Expert—1167
ARC (AI2) Challenge92.2%—
MMLU78.4%—
TriviaQA80%—

Multilingual Not comparable

DeepSeek-V2 (MoE-236B, May 2024): —, Llama 3.1 Nemotron 51b Instruct: 36.3 (#236)

BenchmarkDeepSeek-V2 (MoE-236B, May 2024)Llama 3.1 Nemotron 51b Instruct
LMArena Non-English—1181
LMArena Chinese—1179
LMArena Russian—1186

Instruction Following Not comparable

DeepSeek-V2 (MoE-236B, May 2024): —, Llama 3.1 Nemotron 51b Instruct: 63.2 (#228)

BenchmarkDeepSeek-V2 (MoE-236B, May 2024)Llama 3.1 Nemotron 51b Instruct
LMArena Instruction Following—1201

Long Context Not comparable

DeepSeek-V2 (MoE-236B, May 2024): —, Llama 3.1 Nemotron 51b Instruct: 36.5 (#225)

BenchmarkDeepSeek-V2 (MoE-236B, May 2024)Llama 3.1 Nemotron 51b Instruct
LMArena Longer Query—1205

Writing & Preference Not comparable

DeepSeek-V2 (MoE-236B, May 2024): —, Llama 3.1 Nemotron 51b Instruct: 43.5 (#224)

BenchmarkDeepSeek-V2 (MoE-236B, May 2024)Llama 3.1 Nemotron 51b Instruct
LMArena Text—1228
LMArena Creative Writing—1213
LMArena Multi-Turn—1227

Frequently asked questions

Is DeepSeek-V2 (MoE-236B, May 2024) better than Llama 3.1 Nemotron 51b Instruct?

Llama 3.1 Nemotron 51b Instruct has enough public results to be ranked (#219); DeepSeek-V2 (MoE-236B, May 2024) does not yet, so treat this comparison as directional.

Is DeepSeek-V2 (MoE-236B, May 2024) or Llama 3.1 Nemotron 51b Instruct better for coding?

DeepSeek-V2 (MoE-236B, May 2024) scores higher on coding benchmarks: 39.8 versus 35.0 in the Noometry coding category.

How many benchmarks do DeepSeek-V2 (MoE-236B, May 2024) and Llama 3.1 Nemotron 51b Instruct share?

0 benchmarks have published results for both models. DeepSeek-V2 (MoE-236B, May 2024) has 10 scored results on Noometry and Llama 3.1 Nemotron 51b Instruct has 12.

Related comparisons

Go deeper