Noometry

Model comparison

Claude 3 Haiku vs Phi-4

Phi-4 is the stronger model overall, scoring 31.2 to 25.9 on the Noometry Index.

Last verified . 25 shared benchmarks.

Claude 3 Haiku Anthropic

25.9

Rank #340 Confirmed

Phi-4 Microsoft

31.2

Rank #279 Confirmed

Summary

  • They share 25 benchmarks with published results for both. Claude 3 Haiku scores higher in 1 category and Phi-4 in 7 categories; 6 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Phi-4 leads 32.6 to 17.3.
  • The biggest single-benchmark swing is MATH Level 5: 14.9% for Claude 3 Haiku and 64.9% for Phi-4.
  • Phi-4 has downloadable open weights; the other is API-only.

Side by side

Claude 3 HaikuPhi-4
ProviderAnthropicMicrosoft
Noometry Index25.931.2
Released2024-03-072024-12-11
WeightsProprietaryOpen
Context window—128K
Max output—4K
Input $ / M tokens—$0.07
Output $ / M tokens—$0.14
Results tracked3737

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Phi-4 leads

Claude 3 Haiku: 26.4 (#325), Phi-4: 34.4 (#239)

BenchmarkClaude 3 HaikuPhi-4
BigCodeBench Instruct39.4%45.5%
LMArena Coding11991231
BigCodeBench Complete50.1%55.4%
WeirdML9.8%—
LiveBench Coding—30.7%
CadEval12%—
HumanEval+68.9%—
MBPP+68.8%—

Agentic & Tool Use Not comparable

Claude 3 Haiku: —, Phi-4: 22.8 (#128)

BenchmarkClaude 3 HaikuPhi-4
Berkeley Function Calling Leaderboard—28.8%
BALROG—11.6%

Reasoning Phi-4 leads

Claude 3 Haiku: 16.3 (#307), Phi-4: 17.7 (#291)

BenchmarkClaude 3 HaikuPhi-4
LMArena Hard Prompts11741220
Epoch Capabilities Index118.35130.42
Kagi LLM Benchmark34.2%—
Chess Puzzles—1%
LiveBench Reasoning—47.8%
DTBench50.1%—
LiveBench Data Analysis—45.2%
LMCA8.8%—
ForecastBench53.2—
LiveBench—41.6%
WinoGrande74.2%—

Math Phi-4 leads

Claude 3 Haiku: 9.8 (#319), Phi-4: 20.8 (#285)

BenchmarkClaude 3 HaikuPhi-4
OTIS Mock AIME 2024-20251.8%13.8%
LMArena Math11881246
MATH Level 514.9%64.9%
LiveBench Math—42%

Knowledge Phi-4 leads

Claude 3 Haiku: 17.3 (#285), Phi-4: 32.6 (#209)

BenchmarkClaude 3 HaikuPhi-4
GPQA Diamond36.3%56.1%
Confabulations34.2%29.4%
LMArena Expert11481203
MMLU73.8%84.8%
Vectara Hallucination Rate—3.7%

Multimodal Not comparable

Claude 3 Haiku: 23.6 (#128), Phi-4: —

BenchmarkClaude 3 HaikuPhi-4
LMArena Vision950—
ScienceQA72%—

Multilingual Phi-4 leads

Claude 3 Haiku: 36.0 (#243), Phi-4: 37.2 (#237)

BenchmarkClaude 3 HaikuPhi-4
LMArena Non-English11781197
LMArena Chinese11551212
LMArena French11951224
LMArena German11741222
LMArena Japanese11021158
LMArena Korean11091151
LMArena Russian12041209
LMArena Spanish11661234

Instruction Following Too close to call

Claude 3 Haiku: 61.3 (#247), Phi-4: 60.4 (#251)

BenchmarkClaude 3 HaikuPhi-4
LMArena Instruction Following11731201
LiveBench Instruction Following—58.4%

Long Context Too close to call

Claude 3 Haiku: 36.1 (#237), Phi-4: 36.9 (#226)

BenchmarkClaude 3 HaikuPhi-4
LMArena Longer Query11901217

Writing & Preference Phi-4 leads

Claude 3 Haiku: 29.7 (#291), Phi-4: 40.5 (#244)

BenchmarkClaude 3 HaikuPhi-4
LMArena Text11951217
LMArena Creative Writing11571182
LMArena Multi-Turn11901206
Short-Story Creative Writing—62.6%
EQ-Bench Creative Writing717—
LiveBench Language—25.6%

Frequently asked questions

Is Claude 3 Haiku better than Phi-4?

Phi-4 is the stronger model overall, scoring 31.2 to 25.9 on the Noometry Index.

Is Claude 3 Haiku or Phi-4 better for coding?

Phi-4 scores higher on coding benchmarks: 34.4 versus 26.4 in the Noometry coding category.

How many benchmarks do Claude 3 Haiku and Phi-4 share?

25 benchmarks have published results for both models. Claude 3 Haiku has 37 scored results on Noometry and Phi-4 has 37.

Related comparisons

Go deeper