Noometry

Model comparison

Gemini 1.0 Pro vs GPT-4o mini

Gemini 1.0 Pro is the stronger model overall, scoring 27.3 to 25.5 on the Noometry Index.

Last verified . 24 shared benchmarks.

Gemini 1.0 Pro Google

27.3

Rank #332 Confirmed

GPT-4o mini OpenAI

25.5

Rank #343 Confirmed

Summary

  • They share 24 benchmarks with published results for both. Gemini 1.0 Pro scores higher in 2 categories and GPT-4o mini in 6 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in coding, where Gemini 1.0 Pro leads 32.2 to 22.0.
  • The biggest single-benchmark swing is MATH Level 5: 11.2% for Gemini 1.0 Pro and 52.6% for GPT-4o mini.

Side by side

Gemini 1.0 ProGPT-4o mini
ProviderGoogleOpenAI
Noometry Index27.325.5
Released2023-12-132024-07-18
WeightsProprietaryProprietary
Context window—128K
Max output—16K
Input $ / M tokens—$0.15
Output $ / M tokens—$0.60
Results tracked2460

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemini 1.0 Pro leads

Gemini 1.0 Pro: 32.2 (#275), GPT-4o mini: 22.0 (#335)

BenchmarkGemini 1.0 ProGPT-4o mini
LMArena Coding11081290
HumanEval+55.5%83.5%
MBPP+61.4%72.2%
Aider Polyglot—3.6%
WeirdML—11.8%
BigCodeBench Instruct—46.1%
LiveBench Coding—43.1%
BigCodeBench Complete—57.4%

Agentic & Tool Use Not comparable

Gemini 1.0 Pro: —, GPT-4o mini: 27.5 (#101)

BenchmarkGemini 1.0 ProGPT-4o mini
BALROG—17.4%

Reasoning Gemini 1.0 Pro leads

Gemini 1.0 Pro: 17.1 (#296), GPT-4o mini: 8.7 (#347)

BenchmarkGemini 1.0 ProGPT-4o mini
LMArena Hard Prompts11091267
DTBench45.9%54.4%
Epoch Capabilities Index117.04126.56
ARC-AGI-2—0%
SimpleBench—10.7%
Kagi LLM Benchmark—28.8%
Chess Puzzles—0%
LiveBench Reasoning—32.8%
Mystery Game Puzzles—12%
LiveBench Data Analysis—50%
LMCA—10.4%
LiveBench—41.3%
PIQA—88.7%

Math GPT-4o mini leads

Gemini 1.0 Pro: 9.3 (#321), GPT-4o mini: 10.4 (#314)

BenchmarkGemini 1.0 ProGPT-4o mini
OTIS Mock AIME 2024-20251.1%6.9%
LMArena Math11321267
MATH Level 511.2%52.6%
FrontierMath (Tiers 1-3)—0.7%
Omni-MATH—28%
LiveBench Math—36.3%
GSM8K—91.3%

Knowledge GPT-4o mini leads

Gemini 1.0 Pro: 15.6 (#291), GPT-4o mini: 17.7 (#284)

BenchmarkGemini 1.0 ProGPT-4o mini
GPQA Diamond34%37.7%
LMArena Expert10591235
MMLU70%81.8%
SimpleQA Verified—8.3%
MMLU-Pro—60.3%
Confabulations—37.2%
GPQA (HELM)—36.8%
BoolQ—88.7%

Multimodal Not comparable

Gemini 1.0 Pro: —, GPT-4o mini: 25.9 (#122)

BenchmarkGemini 1.0 ProGPT-4o mini
LMArena Vision—1066
Video-MME—64.8%
GeoBench—64%
VPCT—34%

Multilingual GPT-4o mini leads

Gemini 1.0 Pro: 33.4 (#252), GPT-4o mini: 42.0 (#199)

BenchmarkGemini 1.0 ProGPT-4o mini
LMArena Non-English11381266
LMArena Chinese11241265
LMArena French11451297
LMArena German11251272
LMArena Japanese10231216
LMArena Russian11861275
LMArena Spanish11191276
LMArena Korean—1195

Instruction Following GPT-4o mini leads

Gemini 1.0 Pro: 57.6 (#267), GPT-4o mini: 61.9 (#239)

BenchmarkGemini 1.0 ProGPT-4o mini
LMArena Instruction Following11141258
LiveBench Instruction Following—56.8%
IFEval—78.2%

Long Context GPT-4o mini leads

Gemini 1.0 Pro: 34.3 (#249), GPT-4o mini: 39.1 (#186)

BenchmarkGemini 1.0 ProGPT-4o mini
LMArena Longer Query11321289

Writing & Preference GPT-4o mini leads

Gemini 1.0 Pro: 36.0 (#264), GPT-4o mini: 39.5 (#248)

BenchmarkGemini 1.0 ProGPT-4o mini
LMArena Text11491286
LMArena Creative Writing11311268
LMArena Multi-Turn11391285
Short-Story Creative Writing—67.2%
EQ-Bench Creative Writing—873
WildBench—79.1%
LiveBench Language—28.6%

Frequently asked questions

Is Gemini 1.0 Pro better than GPT-4o mini?

Gemini 1.0 Pro is the stronger model overall, scoring 27.3 to 25.5 on the Noometry Index.

Is Gemini 1.0 Pro or GPT-4o mini better for coding?

Gemini 1.0 Pro scores higher on coding benchmarks: 32.2 versus 22.0 in the Noometry coding category.

How many benchmarks do Gemini 1.0 Pro and GPT-4o mini share?

24 benchmarks have published results for both models. Gemini 1.0 Pro has 24 scored results on Noometry and GPT-4o mini has 60.

Related comparisons

Go deeper