Noometry

Model comparison

Gemini 1.5 Flash (May 2024) vs Gemma 4 26B A4B IT

Gemma 4 26B A4B IT is the stronger model overall, scoring 43.5 to 33.2 on the Noometry Index.

Last verified . 20 shared benchmarks.

Gemini 1.5 Flash (May 2024) Google

33.2

Rank #246 Confirmed

Gemma 4 26B A4B IT Google

43.5

Rank #92 Confirmed

Summary

  • They share 20 benchmarks with published results for both. Gemini 1.5 Flash (May 2024) scores higher in 0 categories and Gemma 4 26B A4B IT in 9 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where Gemma 4 26B A4B IT leads 47.6 to 22.1.
  • The biggest single-benchmark swing is OTIS Mock AIME 2024-2025: 16.3% for Gemini 1.5 Flash (May 2024) and 82.2% for Gemma 4 26B A4B IT.
  • Gemma 4 26B A4B IT has downloadable open weights; the other is API-only.

Side by side

Gemini 1.5 Flash (May 2024)Gemma 4 26B A4B IT
ProviderGoogleGoogle
Noometry Index33.243.5
Released2024-05-142026-04-02
WeightsProprietaryOpen
Context window—262K
Max output—33K
Input $ / M tokens—$0.0675
Output $ / M tokens—$0.23
Results tracked4228

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemma 4 26B A4B IT leads

Gemini 1.5 Flash (May 2024): 34.4 (#236), Gemma 4 26B A4B IT: 39.0 (#164)

BenchmarkGemini 1.5 Flash (May 2024)Gemma 4 26B A4B IT
WeirdML24.9%35.2%
LMArena Coding12611447
LMArena WebDev—1359
SciCode—40%
BigCodeBench Instruct43.5%—
BigCodeBench Complete55.1%—
ALE-Bench—927.17
HumanEval+75.6%—
MBPP+67.5%—

Agentic & Tool Use Not comparable

Gemini 1.5 Flash (May 2024): 26.6 (#102), Gemma 4 26B A4B IT: —

BenchmarkGemini 1.5 Flash (May 2024)Gemma 4 26B A4B IT
BALROG14.6%—

Reasoning Too close to call

Gemini 1.5 Flash (May 2024): 21.7 (#215), Gemma 4 26B A4B IT: 21.8 (#213)

BenchmarkGemini 1.5 Flash (May 2024)Gemma 4 26B A4B IT
LMArena Hard Prompts12571439
DTBench53.8%74.9%
Epoch Capabilities Index129.36141.85
CritPt—0%
Chess Puzzles—6%
LMCA—29.7%
ForecastBench53.9—
PIQA87.5%—

Math Gemma 4 26B A4B IT leads

Gemini 1.5 Flash (May 2024): 22.1 (#281), Gemma 4 26B A4B IT: 47.6 (#67)

BenchmarkGemini 1.5 Flash (May 2024)Gemma 4 26B A4B IT
OTIS Mock AIME 2024-202516.3%82.2%
LMArena Math12691470
Omni-MATH30.4%—
MATH Level 561.9%—
FrontierMath (Feb 2025 set)0%—
GSM8K82.4%—

Knowledge Gemma 4 26B A4B IT leads

Gemini 1.5 Flash (May 2024): 26.2 (#260), Gemma 4 26B A4B IT: 45.7 (#85)

BenchmarkGemini 1.5 Flash (May 2024)Gemma 4 26B A4B IT
GPQA Diamond47.3%73.2%
LMArena Expert12331447
MMLU-Pro67.8%—
Vectara Hallucination Rate—5.2%
GPQA (HELM)43.7%—
BoolQ85.8%—
MMLU77.9%—

Multimodal Gemma 4 26B A4B IT leads

Gemini 1.5 Flash (May 2024): 36.0 (#81), Gemma 4 26B A4B IT: 40.6 (#46)

BenchmarkGemini 1.5 Flash (May 2024)Gemma 4 26B A4B IT
LMArena Vision11411260
Video-MME70.3%—
GeoBench76%—

Multilingual Gemma 4 26B A4B IT leads

Gemini 1.5 Flash (May 2024): 42.9 (#189), Gemma 4 26B A4B IT: 53.1 (#71)

BenchmarkGemini 1.5 Flash (May 2024)Gemma 4 26B A4B IT
LMArena Non-English12781421
LMArena Chinese12951495
LMArena French12581460
LMArena Russian12881434
LMArena Spanish12431417
LMArena German1262—
LMArena Japanese1252—
LMArena Korean1221—

Instruction Following Gemma 4 26B A4B IT leads

Gemini 1.5 Flash (May 2024): 66.8 (#205), Gemma 4 26B A4B IT: 74.8 (#82)

BenchmarkGemini 1.5 Flash (May 2024)Gemma 4 26B A4B IT
LMArena Instruction Following12581420
IFEval83.1%—

Long Context Gemma 4 26B A4B IT leads

Gemini 1.5 Flash (May 2024): 39.0 (#187), Gemma 4 26B A4B IT: 43.6 (#91)

BenchmarkGemini 1.5 Flash (May 2024)Gemma 4 26B A4B IT
LMArena Longer Query12841428

Writing & Preference Gemma 4 26B A4B IT leads

Gemini 1.5 Flash (May 2024): 48.7 (#196), Gemma 4 26B A4B IT: 58.6 (#115)

BenchmarkGemini 1.5 Flash (May 2024)Gemma 4 26B A4B IT
LMArena Text12871434
LMArena Creative Writing12851402
LMArena Multi-Turn12531441
EQ-Bench Creative Writing—1305
WildBench79.2%—

Frequently asked questions

Is Gemini 1.5 Flash (May 2024) better than Gemma 4 26B A4B IT?

Gemma 4 26B A4B IT is the stronger model overall, scoring 43.5 to 33.2 on the Noometry Index.

Is Gemini 1.5 Flash (May 2024) or Gemma 4 26B A4B IT better for coding?

Gemma 4 26B A4B IT scores higher on coding benchmarks: 39.0 versus 34.4 in the Noometry coding category.

How many benchmarks do Gemini 1.5 Flash (May 2024) and Gemma 4 26B A4B IT share?

20 benchmarks have published results for both models. Gemini 1.5 Flash (May 2024) has 42 scored results on Noometry and Gemma 4 26B A4B IT has 28.

Related comparisons

Go deeper