Noometry

Model comparison

Claude 3.5 Haiku vs Phi 3 Mini 128k Instruct

Claude 3.5 Haiku and Phi 3 Mini 128k Instruct score almost the same on the Noometry Index (28.9 vs 29.1), so choose on price, context window or the category you care about most.

Last verified . 19 shared benchmarks.

Claude 3.5 Haiku Anthropic

28.9

Rank #309 Confirmed

Phi 3 Mini 128k Instruct Microsoft

29.1

Rank #303 Confirmed

Summary

  • They share 19 benchmarks with published results for both. Claude 3.5 Haiku scores higher in 5 categories and Phi 3 Mini 128k Instruct in 3 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Claude 3.5 Haiku leads 42.9 to 27.3.
  • The biggest single-benchmark swing is BigCodeBench Complete: 59% for Claude 3.5 Haiku and 40.6% for Phi 3 Mini 128k Instruct.
  • Phi 3 Mini 128k Instruct has downloadable open weights; the other is API-only.

Side by side

Claude 3.5 HaikuPhi 3 Mini 128k Instruct
ProviderAnthropicMicrosoft
Noometry Index28.929.1
Released2024-10-222024-04-23
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked4919

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Claude 3.5 Haiku leads

Claude 3.5 Haiku: 32.6 (#258), Phi 3 Mini 128k Instruct: 28.3 (#307)

BenchmarkClaude 3.5 HaikuPhi 3 Mini 128k Instruct
BigCodeBench Instruct46.1%29.6%
LMArena Coding12871040
BigCodeBench Complete59%40.6%
Aider Polyglot28%—
SciCode27.4%—
WeirdML30.7%—
LiveBench Coding51.4%—
CadEval32%—

Agentic & Tool Use Not comparable

Claude 3.5 Haiku: 27.7 (#90), Phi 3 Mini 128k Instruct: —

BenchmarkClaude 3.5 HaikuPhi 3 Mini 128k Instruct
BALROG19.3%—

Reasoning Phi 3 Mini 128k Instruct leads

Claude 3.5 Haiku: 17.1 (#286), Phi 3 Mini 128k Instruct: 18.7 (#254)

BenchmarkClaude 3.5 HaikuPhi 3 Mini 128k Instruct
LMArena Hard Prompts12511028
CritPt0%—
LiveBench Reasoning28.1%—
DTBench56.7%—
LiveBench Data Analysis48.5%—
Epoch Capabilities Index127.15—
LiveBench43.5%—

Math Phi 3 Mini 128k Instruct leads

Claude 3.5 Haiku: 13.7 (#294), Phi 3 Mini 128k Instruct: 29.2 (#220)

BenchmarkClaude 3.5 HaikuPhi 3 Mini 128k Instruct
LMArena Math12441089
OTIS Mock AIME 2024-20254.3%—
Omni-MATH22.4%—
LiveBench Math35.5%—
MATH Level 546.4%—
FrontierMath (Feb 2025 set)0.3%—

Knowledge Phi 3 Mini 128k Instruct leads

Claude 3.5 Haiku: 18.7 (#276), Phi 3 Mini 128k Instruct: 26.3 (#251)

BenchmarkClaude 3.5 HaikuPhi 3 Mini 128k Instruct
LMArena Expert1208984
GPQA Diamond38.1%—
MMLU-Pro60.5%—
Confabulations36.7%—
GPQA (HELM)36.3%—
MMLU74.3%—

Multimodal Not comparable

Claude 3.5 Haiku: 26.3 (#113), Phi 3 Mini 128k Instruct: —

BenchmarkClaude 3.5 HaikuPhi 3 Mini 128k Instruct
LMArena Vision1092—
GeoBench34%—

Multilingual Claude 3.5 Haiku leads

Claude 3.5 Haiku: 40.2 (#213), Phi 3 Mini 128k Instruct: 25.4 (#280)

BenchmarkClaude 3.5 HaikuPhi 3 Mini 128k Instruct
LMArena Non-English12381000
LMArena Chinese12291016
LMArena French12621038
LMArena German12371006
LMArena Japanese1175899
LMArena Korean1172856
LMArena Russian12531004
LMArena Spanish12611058

Instruction Following Claude 3.5 Haiku leads

Claude 3.5 Haiku: 63.1 (#229), Phi 3 Mini 128k Instruct: 52.1 (#289)

BenchmarkClaude 3.5 HaikuPhi 3 Mini 128k Instruct
LMArena Instruction Following12411023
LiveBench Instruction Following61.9%—
IFEval79.2%—

Long Context Claude 3.5 Haiku leads

Claude 3.5 Haiku: 38.3 (#195), Phi 3 Mini 128k Instruct: 30.3 (#284)

BenchmarkClaude 3.5 HaikuPhi 3 Mini 128k Instruct
LMArena Longer Query1261996

Writing & Preference Claude 3.5 Haiku leads

Claude 3.5 Haiku: 42.9 (#228), Phi 3 Mini 128k Instruct: 27.3 (#296)

BenchmarkClaude 3.5 HaikuPhi 3 Mini 128k Instruct
LMArena Text12551050
LMArena Creative Writing12331024
LMArena Multi-Turn1265989
Short-Story Creative Writing73.5%—
EQ-Bench Creative Writing1146—
WildBench76%—
LiveBench Language35.4%—

Frequently asked questions

Is Claude 3.5 Haiku better than Phi 3 Mini 128k Instruct?

Claude 3.5 Haiku and Phi 3 Mini 128k Instruct score almost the same on the Noometry Index (28.9 vs 29.1), so choose on price, context window or the category you care about most.

Is Claude 3.5 Haiku or Phi 3 Mini 128k Instruct better for coding?

Claude 3.5 Haiku scores higher on coding benchmarks: 32.6 versus 28.3 in the Noometry coding category.

How many benchmarks do Claude 3.5 Haiku and Phi 3 Mini 128k Instruct share?

19 benchmarks have published results for both models. Claude 3.5 Haiku has 49 scored results on Noometry and Phi 3 Mini 128k Instruct has 19.

Related comparisons

Go deeper