Noometry

Model comparison

Mistral Medium vs Phi-4

Mistral Medium is the stronger model overall, scoring 36.0 to 30.9 on the Noometry Index. Phi-4 costs 34× less per token, which makes it the better buy when Mistral Medium's lead doesn't matter for your workload.

Last verified . 23 shared benchmarks.

Mistral Medium Mistral AI

36.0

Rank #211 Confirmed

Phi-4 Microsoft

30.9

Rank #269 Confirmed

Summary

  • They share 23 benchmarks with published results for both. Mistral Medium scores higher in 7 categories and Phi-4 in 2 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Mistral Medium leads 60.2 to 40.7.
  • The biggest single-benchmark swing is Vectara Hallucination Rate: 22.7% for Mistral Medium and 3.7% for Phi-4.
  • Phi-4 is cheaper at $0.07 / $0.14 per million input/output tokens, against $1.50 / $7.50 for Mistral Medium.
  • Mistral Medium accepts more context: 262K tokens versus 128K.

Side by side

Mistral MediumPhi-4
ProviderMistral AIMicrosoft
Noometry Index36.030.9
Released2023-12-112024-12-11
WeightsOpenOpen
Context window262K128K
Max output262K4K
Input $ / M tokens$1.50$0.07
Output $ / M tokens$7.50$0.14
Results tracked3637

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Too close to call

Mistral Medium: 33.8 (#241), Phi-4: 33.8 (#237)

BenchmarkMistral MediumPhi-4
LMArena Coding14341231
FrontierCode8%—
SciCode40.2%—
WeirdML43.7%—
BigCodeBench Instruct—45.5%
LiveBench Coding—30.7%
BigCodeBench Complete—55.4%
ALE-Bench763.98—

Agentic & Tool Use Mistral Medium leads

Mistral Medium: 28.1 (#84), Phi-4: 22.6 (#120)

BenchmarkMistral MediumPhi-4
Berkeley Function Calling Leaderboard37.7%28.8%
BALROG—11.6%

Reasoning Mistral Medium leads

Mistral Medium: 23.3 (#165), Phi-4: 17.0 (#287)

BenchmarkMistral MediumPhi-4
LMArena Hard Prompts14261220
Kagi LLM Benchmark50%—
CritPt0%—
Chess Puzzles—1%
LiveBench Reasoning—47.8%
DTBench75.5%—
LiveBench Data Analysis—45.2%
LMCA26.1%—
Surface Evolver Bench26.9%—
Epoch Capabilities Index—130.43
LiveBench—41.6%

Math Mistral Medium leads

Mistral Medium: 27.2 (#238), Phi-4: 19.6 (#279)

BenchmarkMistral MediumPhi-4
OTIS Mock AIME 2024-202532.2%13.8%
LMArena Math14091246
MATH Level 581.6%64.9%
ProofBench9%—
LiveBench Math—42%
FrontierMath (Feb 2025 set)0.3%—

Knowledge Phi-4 leads

Mistral Medium: 24.9 (#260), Phi-4: 32.5 (#204)

BenchmarkMistral MediumPhi-4
GPQA Diamond59.5%56.1%
Vectara Hallucination Rate22.7%3.7%
LMArena Expert14081203
Humanity's Last Exam4.5%—
Confabulations—29.4%
MMLU—84.8%

Multimodal Not comparable

Mistral Medium: 34.6 (#82), Phi-4: —

BenchmarkMistral MediumPhi-4
LMArena Vision1172—

Multilingual Mistral Medium leads

Mistral Medium: 52.3 (#86), Phi-4: 37.4 (#232)

BenchmarkMistral MediumPhi-4
LMArena Non-English14081197
LMArena Chinese14481211
LMArena French14601223
LMArena German14331222
LMArena Japanese13781158
LMArena Korean13821151
LMArena Russian14111209
LMArena Spanish14331234

Instruction Following Mistral Medium leads

Mistral Medium: 73.9 (#111), Phi-4: 60.6 (#246)

BenchmarkMistral MediumPhi-4
LMArena Instruction Following13981201
LiveBench Instruction Following—58.4%

Long Context Mistral Medium leads

Mistral Medium: 42.9 (#108), Phi-4: 36.9 (#221)

BenchmarkMistral MediumPhi-4
LMArena Longer Query14061217

Writing & Preference Mistral Medium leads

Mistral Medium: 60.2 (#97), Phi-4: 40.7 (#239)

BenchmarkMistral MediumPhi-4
LMArena Text14241217
LMArena Creative Writing13911182
Short-Story Creative Writing77.3%62.6%
LMArena Multi-Turn14181206
LiveBench Language—25.6%

Frequently asked questions

Is Mistral Medium better than Phi-4?

Mistral Medium is the stronger model overall, scoring 36.0 to 30.9 on the Noometry Index. Phi-4 costs 34× less per token, which makes it the better buy when Mistral Medium's lead doesn't matter for your workload.

Which is cheaper, Mistral Medium or Phi-4?

Phi-4 is cheaper. It lists at $0.07 per million input tokens and $0.14 per million output tokens; Mistral Medium lists at $1.50 and $7.50.

Is Mistral Medium or Phi-4 better for coding?

They score almost the same on coding (33.8 vs 33.8); test both on your own repository before choosing.

Which has the bigger context window?

Mistral Medium does, with 262K tokens against 128K.

How many benchmarks do Mistral Medium and Phi-4 share?

23 benchmarks have published results for both models. Mistral Medium has 36 scored results on Noometry and Phi-4 has 37.

Related comparisons

Go deeper