Noometry

Model comparison

Claude 2.1 vs Phi-3.5-mini

Claude 2.1 has enough public results to be ranked (#345); Phi-3.5-mini does not yet, so treat this comparison as directional.

Last verified . 0 shared benchmarks.

Claude 2.1 Anthropic

25.2

Rank #345 Reported

Phi-3.5-mini Microsoft

36.9

Unranked Sparse

Summary

  • The widest gap is in coding, where Phi-3.5-mini leads 34.6 to 26.2.
  • Phi-3.5-mini has downloadable open weights; the other is API-only.

Side by side

Claude 2.1Phi-3.5-mini
ProviderAnthropicMicrosoft
Noometry Index25.236.9
Released2023-11-212024-08-16
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked75

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Phi-3.5-mini leads

Claude 2.1: 26.2 (#327), Phi-3.5-mini: 34.6 (#232)

BenchmarkClaude 2.1Phi-3.5-mini
WeirdML7.1%—
BigCodeBench Instruct—32.8%
BigCodeBench Complete—38.5%

Reasoning Not comparable

Claude 2.1: 21.4 (#221), Phi-3.5-mini: —

BenchmarkClaude 2.1Phi-3.5-mini
DTBench51%—
Epoch Capabilities Index119.27—
ForecastBench54.2—
PIQA—81%

Math Not comparable

Claude 2.1: 10.2 (#315), Phi-3.5-mini: —

BenchmarkClaude 2.1Phi-3.5-mini
OTIS Mock AIME 2024-20251.9%—
GSM8K—86.2%

Knowledge Not comparable

Claude 2.1: 15.4 (#292), Phi-3.5-mini: —

BenchmarkClaude 2.1Phi-3.5-mini
GPQA Diamond33%—
BoolQ—78%
MMLU73.5%—

Frequently asked questions

Is Claude 2.1 better than Phi-3.5-mini?

Claude 2.1 has enough public results to be ranked (#345); Phi-3.5-mini does not yet, so treat this comparison as directional.

Is Claude 2.1 or Phi-3.5-mini better for coding?

Phi-3.5-mini scores higher on coding benchmarks: 34.6 versus 26.2 in the Noometry coding category.

How many benchmarks do Claude 2.1 and Phi-3.5-mini share?

0 benchmarks have published results for both models. Claude 2.1 has 7 scored results on Noometry and Phi-3.5-mini has 5.

Related comparisons

Go deeper