Noometry

Model comparison

Phi 3 Small 8k Instruct vs Qwen2.5-Coder-32B

Qwen2.5-Coder-32B is the stronger model overall, scoring 33.4 to 29.3 on the Noometry Index.

Last verified . 23 shared benchmarks.

Phi 3 Small 8k Instruct Microsoft

29.3

Rank #314 Confirmed

Qwen2.5-Coder-32B Alibaba (Qwen)

33.4

Rank #245 Confirmed

Summary

  • They share 23 benchmarks with published results for both. Phi 3 Small 8k Instruct scores higher in 1 category and Qwen2.5-Coder-32B in 7 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Qwen2.5-Coder-32B leads 41.6 to 31.1.
  • The biggest single-benchmark swing is LiveBench Coding: 20.3% for Phi 3 Small 8k Instruct and 56.9% for Qwen2.5-Coder-32B.

Side by side

Phi 3 Small 8k InstructQwen2.5-Coder-32B
ProviderMicrosoftAlibaba (Qwen)
Noometry Index29.333.4
Released2024-04-232024-09-18
WeightsOpenOpen
Context window—33K
Max output—29K
Input $ / M tokens—$0.66
Output $ / M tokens—$1
Results tracked3231

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Phi 3 Small 8k Instruct leads

Phi 3 Small 8k Instruct: 27.9 (#318), Qwen2.5-Coder-32B: 22.6 (#333)

BenchmarkPhi 3 Small 8k InstructQwen2.5-Coder-32B
LiveBench Coding20.3%56.9%
LMArena Coding11011276
SWE-bench Verified (bash only)—9%
Aider Polyglot—16.4%
BigCodeBench Instruct—49%
BigCodeBench Complete—58%
HumanEval+—87.2%
MBPP+—77%

Reasoning Qwen2.5-Coder-32B leads

Phi 3 Small 8k Instruct: 14.8 (#323), Qwen2.5-Coder-32B: 21.2 (#225)

BenchmarkPhi 3 Small 8k InstructQwen2.5-Coder-32B
LiveBench Reasoning15.9%42.1%
LMArena Hard Prompts11001251
LiveBench Data Analysis30.3%49.9%
HellaSwag77%83%
LiveBench24%46.2%
WinoGrande81.5%80.8%
Adversarial NLI58.1%—
BIG-Bench Hard79.1%—
Epoch Capabilities Index—119.49

Math Qwen2.5-Coder-32B leads

Phi 3 Small 8k Instruct: 27.6 (#248), Qwen2.5-Coder-32B: 33.3 (#204)

BenchmarkPhi 3 Small 8k InstructQwen2.5-Coder-32B
LiveBench Math17.6%46.6%
LMArena Math11511251
GSM8K—93%

Knowledge Qwen2.5-Coder-32B leads

Phi 3 Small 8k Instruct: 29.1 (#240), Qwen2.5-Coder-32B: 33.4 (#203)

BenchmarkPhi 3 Small 8k InstructQwen2.5-Coder-32B
LMArena Expert10671221
ARC (AI2) Challenge90.7%70.5%
MMLU75.7%79.1%
OpenBookQA88%—
TriviaQA58.1%—

Multilingual Qwen2.5-Coder-32B leads

Phi 3 Small 8k Instruct: 28.5 (#272), Qwen2.5-Coder-32B: 37.8 (#235)

BenchmarkPhi 3 Small 8k InstructQwen2.5-Coder-32B
LMArena Non-English10581205
LMArena Chinese10611222
LMArena Russian11111228
LMArena French1135—
LMArena German1080—
LMArena Japanese966—
LMArena Korean894—
LMArena Spanish1111—

Instruction Following Qwen2.5-Coder-32B leads

Phi 3 Small 8k Instruct: 51.9 (#292), Qwen2.5-Coder-32B: 61.4 (#245)

BenchmarkPhi 3 Small 8k InstructQwen2.5-Coder-32B
LiveBench Instruction Following47.2%58.7%
LMArena Instruction Following10871223

Long Context Qwen2.5-Coder-32B leads

Phi 3 Small 8k Instruct: 33.0 (#267), Qwen2.5-Coder-32B: 38.0 (#208)

BenchmarkPhi 3 Small 8k InstructQwen2.5-Coder-32B
LMArena Longer Query10881251

Writing & Preference Qwen2.5-Coder-32B leads

Phi 3 Small 8k Instruct: 31.1 (#284), Qwen2.5-Coder-32B: 41.6 (#240)

BenchmarkPhi 3 Small 8k InstructQwen2.5-Coder-32B
LMArena Text11101230
LMArena Creative Writing10831174
LMArena Multi-Turn10681222
LiveBench Language12.9%23.3%

Frequently asked questions

Is Phi 3 Small 8k Instruct better than Qwen2.5-Coder-32B?

Qwen2.5-Coder-32B is the stronger model overall, scoring 33.4 to 29.3 on the Noometry Index.

Is Phi 3 Small 8k Instruct or Qwen2.5-Coder-32B better for coding?

Phi 3 Small 8k Instruct scores higher on coding benchmarks: 27.9 versus 22.6 in the Noometry coding category.

How many benchmarks do Phi 3 Small 8k Instruct and Qwen2.5-Coder-32B share?

23 benchmarks have published results for both models. Phi 3 Small 8k Instruct has 32 scored results on Noometry and Qwen2.5-Coder-32B has 31.

Related comparisons

Go deeper