Noometry

Model comparison

Gemini 1.0 Pro vs Mistral Small

Mistral Small is the stronger model overall, scoring 33.4 to 27.3 on the Noometry Index.

Last verified . 21 shared benchmarks.

Gemini 1.0 Pro Google

27.3

Rank #332 Confirmed

Mistral Small Mistral AI

33.4

Rank #243 Confirmed

Summary

  • They share 21 benchmarks with published results for both. Gemini 1.0 Pro scores higher in 0 categories and Mistral Small in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Mistral Small leads 52.5 to 36.0.
  • The biggest single-benchmark swing is MATH Level 5: 11.2% for Gemini 1.0 Pro and 46.8% for Mistral Small.
  • Mistral Small has downloadable open weights; the other is API-only.

Side by side

Gemini 1.0 ProMistral Small
ProviderGoogleMistral AI
Noometry Index27.333.4
Released2023-12-132024-02-26
WeightsProprietaryOpen
Context window—262K
Max output—256K
Input $ / M tokens—$0.15
Output $ / M tokens—$0.60
Results tracked2439

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mistral Small leads

Gemini 1.0 Pro: 32.2 (#275), Mistral Small: 34.0 (#247)

BenchmarkGemini 1.0 ProMistral Small
LMArena Coding11081362
SciCode—26.5%
BigCodeBench Instruct—36.1%
LiveBench Coding—36.2%
BigCodeBench Complete—46.6%
ALE-Bench—497.62
HumanEval+55.5%—
MBPP+61.4%—

Agentic & Tool Use Not comparable

Gemini 1.0 Pro: —, Mistral Small: 28.1 (#93)

BenchmarkGemini 1.0 ProMistral Small
Berkeley Function Calling Leaderboard—37.1%

Reasoning Mistral Small leads

Gemini 1.0 Pro: 17.1 (#296), Mistral Small: 19.8 (#250)

BenchmarkGemini 1.0 ProMistral Small
LMArena Hard Prompts11091335
DTBench45.9%70.9%
Kagi LLM Benchmark—37.8%
CritPt—0%
LiveBench Reasoning—44.8%
LiveBench Data Analysis—53.7%
LMCA—20.6%
Epoch Capabilities Index117.04—
LiveBench—44%

Math Mistral Small leads

Gemini 1.0 Pro: 9.3 (#321), Mistral Small: 16.4 (#293)

BenchmarkGemini 1.0 ProMistral Small
OTIS Mock AIME 2024-20251.1%5.8%
LMArena Math11321341
MATH Level 511.2%46.8%
LiveBench Math—39.9%

Knowledge Mistral Small leads

Gemini 1.0 Pro: 15.6 (#291), Mistral Small: 31.0 (#222)

BenchmarkGemini 1.0 ProMistral Small
GPQA Diamond34%47.5%
LMArena Expert10591291
MMLU70%68.7%
Vectara Hallucination Rate—5.1%

Multimodal Not comparable

Gemini 1.0 Pro: —, Mistral Small: 33.5 (#96)

BenchmarkGemini 1.0 ProMistral Small
LMArena Vision—1142

Multilingual Mistral Small leads

Gemini 1.0 Pro: 33.4 (#252), Mistral Small: 45.5 (#169)

BenchmarkGemini 1.0 ProMistral Small
LMArena Non-English11381315
LMArena Chinese11241340
LMArena French11451337
LMArena German11251340
LMArena Japanese10231275
LMArena Russian11861324
LMArena Spanish11191346
LMArena Korean—1259

Instruction Following Mistral Small leads

Gemini 1.0 Pro: 57.6 (#267), Mistral Small: 66.4 (#209)

BenchmarkGemini 1.0 ProMistral Small
LMArena Instruction Following11141310
LiveBench Instruction Following—63.7%

Long Context Mistral Small leads

Gemini 1.0 Pro: 34.3 (#249), Mistral Small: 40.4 (#156)

BenchmarkGemini 1.0 ProMistral Small
LMArena Longer Query11321327

Writing & Preference Mistral Small leads

Gemini 1.0 Pro: 36.0 (#264), Mistral Small: 52.5 (#171)

BenchmarkGemini 1.0 ProMistral Small
LMArena Text11491338
LMArena Creative Writing11311305
LMArena Multi-Turn11391344
LiveBench Language—30.5%

Frequently asked questions

Is Gemini 1.0 Pro better than Mistral Small?

Mistral Small is the stronger model overall, scoring 33.4 to 27.3 on the Noometry Index.

Is Gemini 1.0 Pro or Mistral Small better for coding?

Mistral Small scores higher on coding benchmarks: 34.0 versus 32.2 in the Noometry coding category.

How many benchmarks do Gemini 1.0 Pro and Mistral Small share?

21 benchmarks have published results for both models. Gemini 1.0 Pro has 24 scored results on Noometry and Mistral Small has 39.

Related comparisons

Go deeper