Noometry

Model comparison

Gemini 2.5 Pro vs GPT-5.4

GPT-5.4 is the stronger model overall, scoring 59.4 to 45.0 on the Noometry Index. Gemini 2.5 Pro costs 1.6× less per token, which makes it the better buy when GPT-5.4's lead doesn't matter for your workload.

Last verified . 51 shared benchmarks.

Gemini 2.5 Pro Google

45.0

Rank #75 Confirmed

GPT-5.4 OpenAI

59.4

Rank #16 Confirmed

Summary

  • They share 51 benchmarks with published results for both. Gemini 2.5 Pro scores higher in 2 categories and GPT-5.4 in 8 categories; 9 gaps are clear of the uncertainty.
  • The widest gap is in math, where GPT-5.4 leads 73.5 to 32.5.
  • The biggest single-benchmark swing is ARC-AGI-2: 4.9% for Gemini 2.5 Pro and 74% for GPT-5.4.
  • Gemini 2.5 Pro is cheaper at $1.25 / $10 per million input/output tokens, against $2.50 / $15 for GPT-5.4.
  • GPT-5.4 accepts more context: 1.05M tokens versus 1.05M.

Side by side

Gemini 2.5 ProGPT-5.4
ProviderGoogleOpenAI
Noometry Index45.059.4
Released2025-03-252026-03-05
WeightsProprietaryProprietary
Context window1.05M1.05M
Max output66K128K
Input $ / M tokens$1.25$2.50
Output $ / M tokens$10$15
Results tracked7868

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding GPT-5.4 leads

Gemini 2.5 Pro: 42.4 (#101), GPT-5.4: 52.6 (#33)

BenchmarkGemini 2.5 ProGPT-5.4
SWE-bench Verified57.6%76.9%
LMArena WebDev12271465
SciCode42.8%56.6%
GSO3.9%31.4%
WeirdML54%77.7%
LMArena Coding14521497
ALE-Bench785.521,607
AlgoTune1.511.85
DeepSWE—51.8%
SWE-bench Verified (bash only)53.6%—
Aider Polyglot83.1%—
LiveBench Coding85.9%—
MirrorCode—15.6%
CadEval64%—

Agentic & Tool Use GPT-5.4 leads

Gemini 2.5 Pro: 29.2 (#88), GPT-5.4: 46.5 (#13)

BenchmarkGemini 2.5 ProGPT-5.4
Terminal-Bench32.6%81.8%
τ²-bench Banking13.7%39.4%
DeepResearch Bench42.8%35.1%
LMArena Search11421197
METR Time Horizons55.4%74.3%
Vending-Bench 2573.646,144
APEX-Agents—52.4%
GDPval23.3%—
Remote Labor Index0.8%—
TheAgentCompany30.3%—
PostTrainBench—19%
BALROG43.3%—
GBAEval—45.1%

Reasoning GPT-5.4 leads

Gemini 2.5 Pro: 28.8 (#99), GPT-5.4: 61.8 (#19)

BenchmarkGemini 2.5 ProGPT-5.4
ARC-AGI-24.9%74%
Kagi LLM Benchmark70.3%63.8%
ARC-AGI-141%93.7%
CritPt2%23.4%
Chess Puzzles20%44%
EnigmaEval5.6%16%
LMArena Hard Prompts14551485
DTBench82.4%94.4%
LMCA34.8%52%
Epoch Capabilities Index145.32156.81
ForecastBench61.359.5
SimpleBench62.4%—
NYT Connections (extended)—91.3%
Thematic Generalization—80%
EBR-Bench—25.4%
LiveBench Reasoning89.8%—
Mystery Game Puzzles—37%
LiveBench Data Analysis79.9%—
LiveBench82.3%—

Math GPT-5.4 leads

Gemini 2.5 Pro: 32.5 (#213), GPT-5.4: 73.5 (#19)

BenchmarkGemini 2.5 ProGPT-5.4
FrontierMath (Tiers 1-3)24.6%78.6%
FrontierMath Tier 40%49%
OTIS Mock AIME 2024-202584.7%97.8%
LMArena Math14501488
FrontierMath (Feb 2025 set)14.1%47.6%
FrontierMath Tier 4 (v1)4.2%27.1%
MathArena Final-Answer Competitions—83.1%
ProofBench—56%
Omni-MATH41.6%—
LiveBench Math90.2%—
MATH Level 595.9%—

Knowledge GPT-5.4 leads

Gemini 2.5 Pro: 56.0 (#46), GPT-5.4: 65.3 (#14)

BenchmarkGemini 2.5 ProGPT-5.4
GPQA Diamond85.3%93.3%
Humanity's Last Exam21.6%36.2%
Vectara Hallucination Rate7%7%
LMArena Expert14521507
SimpleQA Verified—45.1%
MMLU-Pro86.3%—
Confabulations10.6%—
GPQA (HELM)74.9%—

Multimodal Gemini 2.5 Pro leads

Gemini 2.5 Pro: 45.2 (#18), GPT-5.4: 43.7 (#20)

BenchmarkGemini 2.5 ProGPT-5.4
LMArena Vision12631303
LMArena Document14211471
GeoBench86%—
VPCT48%—
Blueprint-Bench 2—27.1%
Furniture Assembly—37.5%
SpatialViz-Bench44.7%—

Multilingual Too close to call

Gemini 2.5 Pro: 55.3 (#31), GPT-5.4: 56.2 (#23)

BenchmarkGemini 2.5 ProGPT-5.4
LMArena Non-English14511465
LMArena Chinese15071519
LMArena French14721493
LMArena German14871472
LMArena Japanese14611485
LMArena Korean14341448
LMArena Russian14611480
LMArena Spanish14731454

Instruction Following GPT-5.4 leads

Gemini 2.5 Pro: 75.0 (#75), GPT-5.4: 77.1 (#27)

BenchmarkGemini 2.5 ProGPT-5.4
LMArena Instruction Following14371469
LiveBench Instruction Following80.6%—
IFEval84%—

Long Context Gemini 2.5 Pro leads

Gemini 2.5 Pro: 59.8 (#5), GPT-5.4: 50.3 (#8)

BenchmarkGemini 2.5 ProGPT-5.4
LMArena Longer Query14491473
Fiction.LiveBench91.7%—
CL-bench—27.9%
CL-bench Life—21.7%

Writing & Preference GPT-5.4 leads

Gemini 2.5 Pro: 63.7 (#62), GPT-5.4: 71.9 (#17)

BenchmarkGemini 2.5 ProGPT-5.4
LMArena Text14581469
LMArena Creative Writing14541439
EQ-Bench Creative Writing14211840
LMArena Multi-Turn14531482
Short-Story Creative Writing83.8%—
WildBench85.7%—
EQ-Bench 4—1272
LiveBench Language67.8%—

Frequently asked questions

Is Gemini 2.5 Pro better than GPT-5.4?

GPT-5.4 is the stronger model overall, scoring 59.4 to 45.0 on the Noometry Index. Gemini 2.5 Pro costs 1.6× less per token, which makes it the better buy when GPT-5.4's lead doesn't matter for your workload.

Which is cheaper, Gemini 2.5 Pro or GPT-5.4?

Gemini 2.5 Pro is cheaper. It lists at $1.25 per million input tokens and $10 per million output tokens; GPT-5.4 lists at $2.50 and $15.

Is Gemini 2.5 Pro or GPT-5.4 better for coding?

GPT-5.4 scores higher on coding benchmarks: 52.6 versus 42.4 in the Noometry coding category.

Which has the bigger context window?

GPT-5.4 does, with 1.05M tokens against 1.05M.

How many benchmarks do Gemini 2.5 Pro and GPT-5.4 share?

51 benchmarks have published results for both models. Gemini 2.5 Pro has 78 scored results on Noometry and GPT-5.4 has 68.

Related comparisons

Go deeper