Noometry

Model comparison

Deepseek Coder v2 vs Phi 3 Mini 4k Instruct

Deepseek Coder v2 is the stronger model overall, scoring 35.9 to 27.9 on the Noometry Index.

Last verified . 21 shared benchmarks.

Deepseek Coder v2 DeepSeek

35.9

Rank #220 Confirmed

Phi 3 Mini 4k Instruct Microsoft

27.9

Rank #328 Confirmed

Summary

  • They share 21 benchmarks with published results for both. Deepseek Coder v2 scores higher in 8 categories and Phi 3 Mini 4k Instruct in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in instruction following, where Deepseek Coder v2 leads 61.7 to 47.7.

Side by side

Deepseek Coder v2Phi 3 Mini 4k Instruct
ProviderDeepSeekMicrosoft
Noometry Index35.927.9
Released2024-06-172024-04-23
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked2435

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Deepseek Coder v2 leads

Deepseek Coder v2: 38.1 (#183), Phi 3 Mini 4k Instruct: 26.6 (#323)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Coding12511093
HumanEval+82.3%59.1%
MBPP+75.1%54.2%
BigCodeBench Instruct48.2%—
LiveBench Coding—15.5%
BigCodeBench Complete59.7%—

Reasoning Deepseek Coder v2 leads

Deepseek Coder v2: 23.6 (#176), Phi 3 Mini 4k Instruct: 14.1 (#328)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Hard Prompts12071072
WinoGrande83.7%70.8%
Chess Puzzles—0%
LiveBench Reasoning—26.8%
LiveBench Data Analysis—34.7%
Adversarial NLI—52.8%
BIG-Bench Hard—71.7%
HellaSwag—76.7%
LiveBench—22.4%

Math Deepseek Coder v2 leads

Deepseek Coder v2: 34.9 (#190), Phi 3 Mini 4k Instruct: 26.6 (#257)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Math12411111
LiveBench Math—15.7%
GSM8K94.5%—

Knowledge Deepseek Coder v2 leads

Deepseek Coder v2: 32.3 (#212), Phi 3 Mini 4k Instruct: 28.5 (#246)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Expert11811045
ARC (AI2) Challenge64.3%84.9%
MMLU—68.8%
OpenBookQA—88%
TriviaQA—64%

Multilingual Deepseek Coder v2 leads

Deepseek Coder v2: 36.3 (#240), Phi 3 Mini 4k Instruct: 26.3 (#280)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Non-English11821021
LMArena Chinese12011021
LMArena French11851076
LMArena German11641044
LMArena Japanese1126935
LMArena Korean1104905
LMArena Russian11881022
LMArena Spanish11531085

Instruction Following Deepseek Coder v2 leads

Deepseek Coder v2: 61.7 (#242), Phi 3 Mini 4k Instruct: 47.7 (#303)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Instruction Following11801053
LiveBench Instruction Following—39.1%

Long Context Deepseek Coder v2 leads

Deepseek Coder v2: 37.0 (#224), Phi 3 Mini 4k Instruct: 31.7 (#276)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Longer Query12191044

Writing & Preference Deepseek Coder v2 leads

Deepseek Coder v2: 38.2 (#253), Phi 3 Mini 4k Instruct: 27.6 (#300)

BenchmarkDeepseek Coder v2Phi 3 Mini 4k Instruct
LMArena Text11911073
LMArena Creative Writing11201037
LMArena Multi-Turn11771018
LiveBench Language—9.2%

Frequently asked questions

Is Deepseek Coder v2 better than Phi 3 Mini 4k Instruct?

Deepseek Coder v2 is the stronger model overall, scoring 35.9 to 27.9 on the Noometry Index.

Is Deepseek Coder v2 or Phi 3 Mini 4k Instruct better for coding?

Deepseek Coder v2 scores higher on coding benchmarks: 38.1 versus 26.6 in the Noometry coding category.

How many benchmarks do Deepseek Coder v2 and Phi 3 Mini 4k Instruct share?

21 benchmarks have published results for both models. Deepseek Coder v2 has 24 scored results on Noometry and Phi 3 Mini 4k Instruct has 35.

Related comparisons

Go deeper