Noometry

Model comparison

Claude 3 Sonnet vs phi-3-medium 14B

Claude 3 Sonnet and phi-3-medium 14B score almost the same on the Noometry Index (29.0 vs 29.7), so choose on price, context window or the category you care about most.

Last verified . 7 shared benchmarks.

Claude 3 Sonnet Anthropic

29.0

Rank #319 Confirmed

phi-3-medium 14B Microsoft

29.7

Rank #306 Reported

Summary

  • They share 7 benchmarks with published results for both. Claude 3 Sonnet scores higher in 1 category and phi-3-medium 14B in 2 categories; 3 gaps are clear of the uncertainty.
  • The widest gap is in math, where phi-3-medium 14B leads 27.3 to 10.7.
  • The biggest single-benchmark swing is GPQA Diamond: 40.6% for Claude 3 Sonnet and 27.6% for phi-3-medium 14B.
  • phi-3-medium 14B has downloadable open weights; the other is API-only.

Side by side

Claude 3 Sonnetphi-3-medium 14B
ProviderAnthropicMicrosoft
Noometry Index29.029.7
Released2024-02-292024-04-23
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked3013

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding phi-3-medium 14B leads

Claude 3 Sonnet: 29.6 (#302), phi-3-medium 14B: 36.8 (#201)

BenchmarkClaude 3 Sonnetphi-3-medium 14B
BigCodeBench Instruct42.7%37.6%
BigCodeBench Complete53.8%48.7%
WeirdML10.2%—
LMArena Coding1223—
HumanEval+64%—
MBPP+69.3%—

Reasoning Not comparable

Claude 3 Sonnet: 20.5 (#237), phi-3-medium 14B: —

BenchmarkClaude 3 Sonnetphi-3-medium 14B
Epoch Capabilities Index120.7121.23
WinoGrande75.1%81.5%
LMArena Hard Prompts1197—
DTBench53.6%—
Adversarial NLI—55.8%
BIG-Bench Hard—81.4%
HellaSwag—82.4%

Math phi-3-medium 14B leads

Claude 3 Sonnet: 10.7 (#310), phi-3-medium 14B: 27.3 (#250)

BenchmarkClaude 3 Sonnetphi-3-medium 14B
MATH Level 518.2%17.6%
OTIS Mock AIME 2024-20252.5%—
LMArena Math1213—

Knowledge Claude 3 Sonnet leads

Claude 3 Sonnet: 21.1 (#276), phi-3-medium 14B: 9.1 (#306)

BenchmarkClaude 3 Sonnetphi-3-medium 14B
GPQA Diamond40.6%27.6%
MMLU75.9%78%
LMArena Expert1173—
ARC (AI2) Challenge—91.6%
OpenBookQA—87.4%
TriviaQA—73.9%

Multimodal Not comparable

Claude 3 Sonnet: 25.2 (#125), phi-3-medium 14B: —

BenchmarkClaude 3 Sonnetphi-3-medium 14B
LMArena Vision984—

Multilingual Not comparable

Claude 3 Sonnet: 37.8 (#234), phi-3-medium 14B: —

BenchmarkClaude 3 Sonnetphi-3-medium 14B
LMArena Non-English1205—
LMArena Chinese1189—
LMArena French1229—
LMArena German1204—
LMArena Japanese1131—
LMArena Korean1128—
LMArena Russian1227—
LMArena Spanish1204—

Instruction Following Not comparable

Claude 3 Sonnet: 62.8 (#235), phi-3-medium 14B: —

BenchmarkClaude 3 Sonnetphi-3-medium 14B
LMArena Instruction Following1199—

Long Context Not comparable

Claude 3 Sonnet: 36.7 (#228), phi-3-medium 14B: —

BenchmarkClaude 3 Sonnetphi-3-medium 14B
LMArena Longer Query1211—

Writing & Preference Not comparable

Claude 3 Sonnet: 42.1 (#238), phi-3-medium 14B: —

BenchmarkClaude 3 Sonnetphi-3-medium 14B
LMArena Text1218—
LMArena Creative Writing1186—
LMArena Multi-Turn1227—

Frequently asked questions

Is Claude 3 Sonnet better than phi-3-medium 14B?

Claude 3 Sonnet and phi-3-medium 14B score almost the same on the Noometry Index (29.0 vs 29.7), so choose on price, context window or the category you care about most.

Is Claude 3 Sonnet or phi-3-medium 14B better for coding?

phi-3-medium 14B scores higher on coding benchmarks: 36.8 versus 29.6 in the Noometry coding category.

How many benchmarks do Claude 3 Sonnet and phi-3-medium 14B share?

7 benchmarks have published results for both models. Claude 3 Sonnet has 30 scored results on Noometry and phi-3-medium 14B has 13.

Related comparisons

Go deeper