Noometry

Model comparison

Mistral Small vs o1-mini

Mistral Small and o1-mini score almost the same on the Noometry Index (33.4 vs 34.0), so choose on price, context window or the category you care about most.

Last verified . 27 shared benchmarks.

Mistral Small Mistral AI

33.4

Rank #243 Confirmed

o1-mini OpenAI

34.0

Rank #235 Confirmed

Summary

  • They share 27 benchmarks with published results for both. Mistral Small scores higher in 5 categories and o1-mini in 4 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in math, where o1-mini leads 35.4 to 16.4.
  • The biggest single-benchmark swing is MATH Level 5: 46.8% for Mistral Small and 89.2% for o1-mini.
  • Mistral Small has downloadable open weights; the other is API-only.

Side by side

Mistral Smallo1-mini
ProviderMistral AIOpenAI
Noometry Index33.434.0
Released2024-02-262024-09-12
WeightsOpenProprietary
Context window262K—
Max output256K—
Input $ / M tokens$0.15—
Output $ / M tokens$0.60—
Results tracked3939

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o1-mini leads

Mistral Small: 34.0 (#247), o1-mini: 35.5 (#224)

BenchmarkMistral Smallo1-mini
LiveBench Coding36.2%48%
LMArena Coding13621362
Aider Polyglot—32.9%
SciCode26.5%—
WeirdML—36.3%
BigCodeBench Instruct36.1%—
BigCodeBench Complete46.6%—
ALE-Bench497.62—
HumanEval+—89%
MBPP+—78.8%

Agentic & Tool Use Mistral Small leads

Mistral Small: 28.1 (#93), o1-mini: 24.6 (#118)

BenchmarkMistral Smallo1-mini
Berkeley Function Calling Leaderboard37.1%—
Cybench—10%

Reasoning Mistral Small leads

Mistral Small: 19.8 (#250), o1-mini: 8.8 (#346)

BenchmarkMistral Smallo1-mini
LiveBench Reasoning44.8%72.3%
LMArena Hard Prompts13351333
LiveBench Data Analysis53.7%57.9%
LiveBench44%57.8%
ARC-AGI-2—0.8%
SimpleBench—18.1%
Kagi LLM Benchmark37.8%—
ARC-AGI-1—14%
CritPt0%—
DTBench70.9%—
LMCA20.6%—
Epoch Capabilities Index—135.82

Math o1-mini leads

Mistral Small: 16.4 (#293), o1-mini: 35.4 (#186)

BenchmarkMistral Smallo1-mini
OTIS Mock AIME 2024-20255.8%46.9%
LiveBench Math39.9%62%
LMArena Math13411358
MATH Level 546.8%89.2%
FrontierMath (Feb 2025 set)—1.7%

Knowledge o1-mini leads

Mistral Small: 31.0 (#222), o1-mini: 34.9 (#192)

BenchmarkMistral Smallo1-mini
GPQA Diamond47.5%62.4%
LMArena Expert12911316
Confabulations—18.6%
Vectara Hallucination Rate5.1%—
MMLU68.7%—

Multimodal Not comparable

Mistral Small: 33.5 (#96), o1-mini: —

BenchmarkMistral Smallo1-mini
LMArena Vision1142—

Multilingual Mistral Small leads

Mistral Small: 45.5 (#169), o1-mini: 43.6 (#182)

BenchmarkMistral Smallo1-mini
LMArena Non-English13151289
LMArena Chinese13401314
LMArena French13371293
LMArena German13401278
LMArena Japanese12751245
LMArena Korean12591223
LMArena Russian13241283
LMArena Spanish13461303

Instruction Following Too close to call

Mistral Small: 66.4 (#209), o1-mini: 66.7 (#206)

BenchmarkMistral Smallo1-mini
LiveBench Instruction Following63.7%65.4%
LMArena Instruction Following13101304

Long Context Too close to call

Mistral Small: 40.4 (#156), o1-mini: 40.1 (#161)

BenchmarkMistral Smallo1-mini
LMArena Longer Query13271320

Writing & Preference Mistral Small leads

Mistral Small: 52.5 (#171), o1-mini: 48.4 (#202)

BenchmarkMistral Smallo1-mini
LMArena Text13381317
LMArena Creative Writing13051244
LMArena Multi-Turn13441314
LiveBench Language30.5%40.9%
Short-Story Creative Writing—64.9%

Frequently asked questions

Is Mistral Small better than o1-mini?

Mistral Small and o1-mini score almost the same on the Noometry Index (33.4 vs 34.0), so choose on price, context window or the category you care about most.

Is Mistral Small or o1-mini better for coding?

o1-mini scores higher on coding benchmarks: 35.5 versus 34.0 in the Noometry coding category.

How many benchmarks do Mistral Small and o1-mini share?

27 benchmarks have published results for both models. Mistral Small has 39 scored results on Noometry and o1-mini has 39.

Related comparisons

Go deeper