Noometry

Model comparison

Deepseek Coder v2 vs Mixtral 8x22B

Deepseek Coder v2 is the stronger model overall, scoring 35.9 to 27.1 on the Noometry Index.

Last verified . 21 shared benchmarks.

Deepseek Coder v2 DeepSeek

35.9

Rank #220 Confirmed

Mixtral 8x22B Mistral AI

27.1

Rank #333 Confirmed

Summary

  • They share 21 benchmarks with published results for both. Deepseek Coder v2 scores higher in 8 categories and Mixtral 8x22B in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Deepseek Coder v2 leads 32.3 to 15.1.
  • The biggest single-benchmark swing is BigCodeBench Complete: 59.7% for Deepseek Coder v2 and 50.2% for Mixtral 8x22B.

Side by side

Deepseek Coder v2Mixtral 8x22B
ProviderDeepSeekMistral AI
Noometry Index35.927.1
Released2024-06-172024-04-17
WeightsOpenOpen
Context window—64K
Max output—64K
Input $ / M tokens—$2
Output $ / M tokens—$6
Results tracked2434

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Deepseek Coder v2 leads

Deepseek Coder v2: 38.1 (#183), Mixtral 8x22B: 24.2 (#329)

BenchmarkDeepseek Coder v2Mixtral 8x22B
BigCodeBench Instruct48.2%40.6%
LMArena Coding12511166
BigCodeBench Complete59.7%50.2%
HumanEval+82.3%72%
MBPP+75.1%64.3%
WeirdML—3.2%

Agentic & Tool Use Not comparable

Deepseek Coder v2: —, Mixtral 8x22B: 23.1 (#127)

BenchmarkDeepseek Coder v2Mixtral 8x22B
Cybench—7.5%

Reasoning Deepseek Coder v2 leads

Deepseek Coder v2: 23.6 (#176), Mixtral 8x22B: 19.9 (#248)

BenchmarkDeepseek Coder v2Mixtral 8x22B
LMArena Hard Prompts12071150
DTBench—55.1%
Epoch Capabilities Index—122.03
ForecastBench—56.3
WinoGrande83.7%—

Math Deepseek Coder v2 leads

Deepseek Coder v2: 34.9 (#190), Mixtral 8x22B: 22.9 (#275)

BenchmarkDeepseek Coder v2Mixtral 8x22B
LMArena Math12411184
Omni-MATH—16.3%
MATH Level 5—24.2%
GSM8K94.5%—

Knowledge Deepseek Coder v2 leads

Deepseek Coder v2: 32.3 (#212), Mixtral 8x22B: 15.1 (#293)

BenchmarkDeepseek Coder v2Mixtral 8x22B
LMArena Expert11811113
GPQA Diamond—34.1%
MMLU-Pro—46%
GPQA (HELM)—33.4%
ARC (AI2) Challenge64.3%—
MMLU—77.8%

Multilingual Deepseek Coder v2 leads

Deepseek Coder v2: 36.3 (#240), Mixtral 8x22B: 32.8 (#255)

BenchmarkDeepseek Coder v2Mixtral 8x22B
LMArena Non-English11821128
LMArena Chinese12011116
LMArena French11851166
LMArena German11641141
LMArena Japanese11261037
LMArena Korean11041057
LMArena Russian11881158
LMArena Spanish11531151

Instruction Following Deepseek Coder v2 leads

Deepseek Coder v2: 61.7 (#242), Mixtral 8x22B: 57.7 (#266)

BenchmarkDeepseek Coder v2Mixtral 8x22B
LMArena Instruction Following11801147
IFEval—72.4%

Long Context Deepseek Coder v2 leads

Deepseek Coder v2: 37.0 (#224), Mixtral 8x22B: 34.7 (#247)

BenchmarkDeepseek Coder v2Mixtral 8x22B
LMArena Longer Query12191144

Writing & Preference Deepseek Coder v2 leads

Deepseek Coder v2: 38.2 (#253), Mixtral 8x22B: 36.9 (#262)

BenchmarkDeepseek Coder v2Mixtral 8x22B
LMArena Text11911162
LMArena Creative Writing11201141
LMArena Multi-Turn11771130
WildBench—71.1%

Frequently asked questions

Is Deepseek Coder v2 better than Mixtral 8x22B?

Deepseek Coder v2 is the stronger model overall, scoring 35.9 to 27.1 on the Noometry Index.

Is Deepseek Coder v2 or Mixtral 8x22B better for coding?

Deepseek Coder v2 scores higher on coding benchmarks: 38.1 versus 24.2 in the Noometry coding category.

How many benchmarks do Deepseek Coder v2 and Mixtral 8x22B share?

21 benchmarks have published results for both models. Deepseek Coder v2 has 24 scored results on Noometry and Mixtral 8x22B has 34.

Related comparisons

Go deeper