Model comparison

Devstral Small 2505 vs Gemma 4 31B IT

Gemma 4 31B IT is the stronger model overall, scoring 43.5 to 34.3 on the Noometry Index.

Last verified October 10, 2026. 3 shared benchmarks.

Devstral Small 2505

Mistral AI

34.3

Rank #233 Reported

Gemma 4 31B IT

Google

43.5

Rank #90 Confirmed

Summary

They share 3 benchmarks with published results for both. Devstral Small 2505 scores higher in 0 categories and Gemma 4 31B IT in 2 categories; 2 gaps are clear of the uncertainty.
The widest gap is in reasoning, where Gemma 4 31B IT leads 27.2 to 19.7.
The biggest single-benchmark swing is Kagi LLM Benchmark: 37.7% for Devstral Small 2505 and 63.5% for Gemma 4 31B IT.
Both cost about the same: $0.10 input and $0.30 output per million tokens.
Gemma 4 31B IT accepts more context: 262K tokens versus 128K.

Side by side

Devstral Small 2505 and Gemma 4 31B IT specifications
	Devstral Small 2505	Gemma 4 31B IT
Provider	Mistral AI	Google
Noometry Index	34.3	43.5
Released	2025-05-07	2026-04-02
Weights	Open	Open
Context window	128K	262K
Max output	128K	33K
Input $ / M tokens	$0.10	$0.09
Output $ / M tokens	$0.30	$0.34
Results tracked	4	35

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemma 4 31B IT leads

Devstral Small 2505: 38.9 (#166), Gemma 4 31B IT: 42.3 (#108)

Coding benchmarks
Benchmark	Devstral Small 2505	Gemma 4 31B IT
SciCode	28.8%	43.4%
SWE-bench Verified (bash only)	56.4%	—
LMArena WebDev	—	1366
WeirdML	—	52.3%
LMArena Coding	—	1459
ALE-Bench	—	925.5

Reasoning Gemma 4 31B IT leads

Devstral Small 2505: 19.7 (#252), Gemma 4 31B IT: 27.2 (#122)

Reasoning benchmarks
Benchmark	Devstral Small 2505	Gemma 4 31B IT
Kagi LLM Benchmark	37.7%	63.5%
CritPt	0%	1.4%
NYT Connections (extended)	—	70.6%
Chess Puzzles	—	5%
Thematic Generalization	—	53%
LMArena Hard Prompts	—	1448
DTBench	—	82.7%
LMCA	—	39.3%
Surface Evolver Bench	—	30.6%
Epoch Capabilities Index	—	142.74

Math Not comparable

Devstral Small 2505: —, Gemma 4 31B IT: 43.2 (#81)

Math benchmarks
Benchmark	Devstral Small 2505	Gemma 4 31B IT
OTIS Mock AIME 2024-2025	—	73.3%
LMArena Math	—	1465

Knowledge Not comparable

Devstral Small 2505: —, Gemma 4 31B IT: 37.9 (#151)

Knowledge benchmarks
Benchmark	Devstral Small 2505	Gemma 4 31B IT
GPQA Diamond	—	75.8%
SimpleQA Verified	—	10.4%
Vectara Hallucination Rate	—	7.4%
LMArena Expert	—	1465

Multimodal Not comparable

Devstral Small 2505: —, Gemma 4 31B IT: 41.6 (#34)

Multimodal benchmarks
Benchmark	Devstral Small 2505	Gemma 4 31B IT
LMArena Vision	—	1277
LMArena Document	—	1425

Multilingual Not comparable

Devstral Small 2505: —, Gemma 4 31B IT: 53.8 (#57)

Multilingual benchmarks
Benchmark	Devstral Small 2505	Gemma 4 31B IT
LMArena Non-English	—	1431
LMArena Chinese	—	1476
LMArena French	—	1435
LMArena Russian	—	1460
LMArena Spanish	—	1444

Instruction Following Not comparable

Devstral Small 2505: —, Gemma 4 31B IT: 75.5 (#61)

Instruction Following benchmarks
Benchmark	Devstral Small 2505	Gemma 4 31B IT
LMArena Instruction Following	—	1433

Long Context Not comparable

Devstral Small 2505: —, Gemma 4 31B IT: 44.2 (#71)

Long Context benchmarks
Benchmark	Devstral Small 2505	Gemma 4 31B IT
LMArena Longer Query	—	1446

Writing & Preference Not comparable

Devstral Small 2505: —, Gemma 4 31B IT: 60.5 (#96)

Writing & Preference benchmarks
Benchmark	Devstral Small 2505	Gemma 4 31B IT
LMArena Text	—	1443
LMArena Creative Writing	—	1415
EQ-Bench Creative Writing	—	1368
EQ-Bench 4	—	1120
LMArena Multi-Turn	—	1452

Frequently asked questions

Is Devstral Small 2505 better than Gemma 4 31B IT?

Gemma 4 31B IT is the stronger model overall, scoring 43.5 to 34.3 on the Noometry Index.

Which is cheaper, Devstral Small 2505 or Gemma 4 31B IT?

Devstral Small 2505 is cheaper. It lists at $0.10 per million input tokens and $0.30 per million output tokens; Gemma 4 31B IT lists at $0.09 and $0.34.

Is Devstral Small 2505 or Gemma 4 31B IT better for coding?

Gemma 4 31B IT scores higher on coding benchmarks: 42.3 versus 38.9 in the Noometry coding category.

Which has the bigger context window?

Gemma 4 31B IT does, with 262K tokens against 128K.

How many benchmarks do Devstral Small 2505 and Gemma 4 31B IT share?

3 benchmarks have published results for both models. Devstral Small 2505 has 4 scored results on Noometry and Gemma 4 31B IT has 35.