Noometry

Model comparison

Mistral Medium 3.5 vs Phi 3 Mini 4k Instruct

Mistral Medium 3.5 is the stronger model overall, scoring 40.2 to 27.9 on the Noometry Index.

Last verified . 16 shared benchmarks.

Mistral Medium 3.5 Mistral AI

40.2

Rank #152 Confirmed

Phi 3 Mini 4k Instruct Microsoft

27.9

Rank #328 Confirmed

Summary

  • They share 16 benchmarks with published results for both. Mistral Medium 3.5 scores higher in 8 categories and Phi 3 Mini 4k Instruct in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Mistral Medium 3.5 leads 58.5 to 27.6.

Side by side

Mistral Medium 3.5Phi 3 Mini 4k Instruct
ProviderMistral AIMicrosoft
Noometry Index40.227.9
Released—2024-04-23
WeightsOpenOpen
Context window262K—
Max output210K—
Input $ / M tokens$1.50—
Output $ / M tokens$7.50—
Results tracked2235

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mistral Medium 3.5 leads

Mistral Medium 3.5: 36.0 (#213), Phi 3 Mini 4k Instruct: 26.6 (#323)

BenchmarkMistral Medium 3.5Phi 3 Mini 4k Instruct
LMArena Coding14611093
LMArena WebDev1264—
LiveBench Coding—15.5%
HumanEval+—59.1%
MBPP+—54.2%

Reasoning Mistral Medium 3.5 leads

Mistral Medium 3.5: 17.3 (#295), Phi 3 Mini 4k Instruct: 14.1 (#328)

BenchmarkMistral Medium 3.5Phi 3 Mini 4k Instruct
LMArena Hard Prompts14361072
Kagi LLM Benchmark41.4%—
NYT Connections (extended)12.9%—
Chess Puzzles—0%
LiveBench Reasoning—26.8%
LiveBench Data Analysis—34.7%
Adversarial NLI—52.8%
BIG-Bench Hard—71.7%
Epoch Capabilities Index141.35—
HellaSwag—76.7%
LiveBench—22.4%
WinoGrande—70.8%

Math Mistral Medium 3.5 leads

Mistral Medium 3.5: 39.1 (#113), Phi 3 Mini 4k Instruct: 26.6 (#257)

BenchmarkMistral Medium 3.5Phi 3 Mini 4k Instruct
LMArena Math14311111
LiveBench Math—15.7%

Knowledge Mistral Medium 3.5 leads

Mistral Medium 3.5: 40.0 (#126), Phi 3 Mini 4k Instruct: 28.5 (#246)

BenchmarkMistral Medium 3.5Phi 3 Mini 4k Instruct
LMArena Expert14321045
ARC (AI2) Challenge—84.9%
MMLU—68.8%
OpenBookQA—88%
TriviaQA—64%

Multimodal Not comparable

Mistral Medium 3.5: 38.3 (#65), Phi 3 Mini 4k Instruct: —

BenchmarkMistral Medium 3.5Phi 3 Mini 4k Instruct
LMArena Vision1223—

Multilingual Mistral Medium 3.5 leads

Mistral Medium 3.5: 51.9 (#100), Phi 3 Mini 4k Instruct: 26.3 (#280)

BenchmarkMistral Medium 3.5Phi 3 Mini 4k Instruct
LMArena Non-English14041021
LMArena Chinese14421021
LMArena French14481076
LMArena German14511044
LMArena Korean1385905
LMArena Russian13951022
LMArena Spanish14091085
LMArena Japanese—935

Instruction Following Mistral Medium 3.5 leads

Mistral Medium 3.5: 74.6 (#90), Phi 3 Mini 4k Instruct: 47.7 (#303)

BenchmarkMistral Medium 3.5Phi 3 Mini 4k Instruct
LMArena Instruction Following14151053
LiveBench Instruction Following—39.1%

Long Context Mistral Medium 3.5 leads

Mistral Medium 3.5: 43.2 (#103), Phi 3 Mini 4k Instruct: 31.7 (#276)

BenchmarkMistral Medium 3.5Phi 3 Mini 4k Instruct
LMArena Longer Query14151044

Writing & Preference Mistral Medium 3.5 leads

Mistral Medium 3.5: 58.5 (#117), Phi 3 Mini 4k Instruct: 27.6 (#300)

BenchmarkMistral Medium 3.5Phi 3 Mini 4k Instruct
LMArena Text14211073
LMArena Creative Writing13741037
LMArena Multi-Turn14231018
EQ-Bench 4993—
LiveBench Language—9.2%

Frequently asked questions

Is Mistral Medium 3.5 better than Phi 3 Mini 4k Instruct?

Mistral Medium 3.5 is the stronger model overall, scoring 40.2 to 27.9 on the Noometry Index.

Is Mistral Medium 3.5 or Phi 3 Mini 4k Instruct better for coding?

Mistral Medium 3.5 scores higher on coding benchmarks: 36.0 versus 26.6 in the Noometry coding category.

How many benchmarks do Mistral Medium 3.5 and Phi 3 Mini 4k Instruct share?

16 benchmarks have published results for both models. Mistral Medium 3.5 has 22 scored results on Noometry and Phi 3 Mini 4k Instruct has 35.

Related comparisons

Go deeper