Noometry

Model comparison

DeepSeek-V2.5 (Sep 2024) vs Mixtral 8x22B

DeepSeek-V2.5 (Sep 2024) is the stronger model overall, scoring 37.6 to 27.1 on the Noometry Index.

Last verified . 21 shared benchmarks.

DeepSeek-V2.5 (Sep 2024) DeepSeek

37.6

Rank #200 Confirmed

Mixtral 8x22B Mistral AI

27.1

Rank #333 Confirmed

Summary

  • They share 21 benchmarks with published results for both. DeepSeek-V2.5 (Sep 2024) scores higher in 8 categories and Mixtral 8x22B in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where DeepSeek-V2.5 (Sep 2024) leads 34.8 to 15.1.
  • The biggest single-benchmark swing is BigCodeBench Instruct: 48.6% for DeepSeek-V2.5 (Sep 2024) and 40.6% for Mixtral 8x22B.

Side by side

DeepSeek-V2.5 (Sep 2024)Mixtral 8x22B
ProviderDeepSeekMistral AI
Noometry Index37.627.1
Released2024-09-062024-04-17
WeightsOpenOpen
Context window—64K
Max output—64K
Input $ / M tokens—$2
Output $ / M tokens—$6
Results tracked2234

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 31.7 (#281), Mixtral 8x22B: 24.2 (#329)

BenchmarkDeepSeek-V2.5 (Sep 2024)Mixtral 8x22B
BigCodeBench Instruct48.6%40.6%
LMArena Coding13091166
BigCodeBench Complete53.2%50.2%
HumanEval+83.5%72%
MBPP+74.1%64.3%
Aider Polyglot17.8%—
WeirdML—3.2%

Agentic & Tool Use Not comparable

DeepSeek-V2.5 (Sep 2024): —, Mixtral 8x22B: 23.1 (#127)

BenchmarkDeepSeek-V2.5 (Sep 2024)Mixtral 8x22B
Cybench—7.5%

Reasoning DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 25.6 (#145), Mixtral 8x22B: 19.9 (#248)

BenchmarkDeepSeek-V2.5 (Sep 2024)Mixtral 8x22B
LMArena Hard Prompts12891150
DTBench—55.1%
Epoch Capabilities Index—122.03
ForecastBench—56.3

Math DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 35.9 (#177), Mixtral 8x22B: 22.9 (#275)

BenchmarkDeepSeek-V2.5 (Sep 2024)Mixtral 8x22B
LMArena Math12881184
Omni-MATH—16.3%
MATH Level 5—24.2%

Knowledge DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 34.8 (#193), Mixtral 8x22B: 15.1 (#293)

BenchmarkDeepSeek-V2.5 (Sep 2024)Mixtral 8x22B
LMArena Expert12661113
GPQA Diamond—34.1%
MMLU-Pro—46%
GPQA (HELM)—33.4%
MMLU—77.8%

Multilingual DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 42.5 (#193), Mixtral 8x22B: 32.8 (#255)

BenchmarkDeepSeek-V2.5 (Sep 2024)Mixtral 8x22B
LMArena Non-English12731128
LMArena Chinese13181116
LMArena French12891166
LMArena German12581141
LMArena Japanese12281037
LMArena Korean12091057
LMArena Russian12891158
LMArena Spanish12481151

Instruction Following DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 67.5 (#194), Mixtral 8x22B: 57.7 (#266)

BenchmarkDeepSeek-V2.5 (Sep 2024)Mixtral 8x22B
LMArena Instruction Following12801147
IFEval—72.4%

Long Context DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 39.5 (#174), Mixtral 8x22B: 34.7 (#247)

BenchmarkDeepSeek-V2.5 (Sep 2024)Mixtral 8x22B
LMArena Longer Query13011144

Writing & Preference DeepSeek-V2.5 (Sep 2024) leads

DeepSeek-V2.5 (Sep 2024): 49.8 (#187), Mixtral 8x22B: 36.9 (#262)

BenchmarkDeepSeek-V2.5 (Sep 2024)Mixtral 8x22B
LMArena Text12941162
LMArena Creative Writing12851141
LMArena Multi-Turn12971130
WildBench—71.1%

Frequently asked questions

Is DeepSeek-V2.5 (Sep 2024) better than Mixtral 8x22B?

DeepSeek-V2.5 (Sep 2024) is the stronger model overall, scoring 37.6 to 27.1 on the Noometry Index.

Is DeepSeek-V2.5 (Sep 2024) or Mixtral 8x22B better for coding?

DeepSeek-V2.5 (Sep 2024) scores higher on coding benchmarks: 31.7 versus 24.2 in the Noometry coding category.

How many benchmarks do DeepSeek-V2.5 (Sep 2024) and Mixtral 8x22B share?

21 benchmarks have published results for both models. DeepSeek-V2.5 (Sep 2024) has 22 scored results on Noometry and Mixtral 8x22B has 34.

Related comparisons

Go deeper