Model comparison

Devstral Small 2505 vs Gemma 3 4B

Devstral Small 2505 is the stronger model overall, scoring 34.3 to 28.1 on the Noometry Index. Gemma 3 4B costs 3.0× less per token, which makes it the better buy when Devstral Small 2505's lead doesn't matter for your workload.

Last verified October 10, 2026. 1 shared benchmarks.

Devstral Small 2505

Mistral AI

34.3

Rank #233 Reported

Gemma 3 4B

Google

28.1

Rank #326 Confirmed

Summary

They share 1 benchmark with published results for both. Devstral Small 2505 scores higher in 2 categories and Gemma 3 4B in 0 categories; 2 gaps are clear of the uncertainty.
The widest gap is in reasoning, where Devstral Small 2505 leads 19.7 to 13.2.
The biggest single-benchmark swing is Kagi LLM Benchmark: 37.7% for Devstral Small 2505 and 25.2% for Gemma 3 4B.
Gemma 3 4B is cheaper at $0.04 / $0.08 per million input/output tokens, against $0.10 / $0.30 for Devstral Small 2505.
Gemma 3 4B accepts more context: 131K tokens versus 128K.

Side by side

Devstral Small 2505 and Gemma 3 4B specifications
	Devstral Small 2505	Gemma 3 4B
Provider	Mistral AI	Google
Noometry Index	34.3	28.1
Released	2025-05-07	2025-03-12
Weights	Open	Open
Context window	128K	131K
Max output	128K	4K
Input $ / M tokens	$0.10	$0.04
Output $ / M tokens	$0.30	$0.08
Results tracked	4	22

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Devstral Small 2505 leads

Devstral Small 2505: 38.9 (#166), Gemma 3 4B: 35.9 (#215)

Coding benchmarks
Benchmark	Devstral Small 2505	Gemma 3 4B
SWE-bench Verified (bash only)	56.4%	—
SciCode	28.8%	—
LMArena Coding	—	1230

Agentic & Tool Use Not comparable

Devstral Small 2505: —, Gemma 3 4B: 20.9 (#142)

Agentic & Tool Use benchmarks
Benchmark	Devstral Small 2505	Gemma 3 4B
Berkeley Function Calling Leaderboard	—	19.6%

Reasoning Devstral Small 2505 leads

Devstral Small 2505: 19.7 (#252), Gemma 3 4B: 13.2 (#335)

Reasoning benchmarks
Benchmark	Devstral Small 2505	Gemma 3 4B
Kagi LLM Benchmark	37.7%	25.2%
CritPt	0%	—
Chess Puzzles	—	0%
LMArena Hard Prompts	—	1253
DTBench	—	50.9%
LMCA	—	2.8%
Epoch Capabilities Index	—	116.02

Math Not comparable

Devstral Small 2505: —, Gemma 3 4B: 16.8 (#292)

Math benchmarks
Benchmark	Devstral Small 2505	Gemma 3 4B
OTIS Mock AIME 2024-2025	—	7.5%
LMArena Math	—	1239

Knowledge Not comparable

Devstral Small 2505: —, Gemma 3 4B: 11.8 (#299)

Knowledge benchmarks
Benchmark	Devstral Small 2505	Gemma 3 4B
GPQA Diamond	—	23.2%
Vectara Hallucination Rate	—	6.4%
LMArena Expert	—	1223

Multilingual Not comparable

Devstral Small 2505: —, Gemma 3 4B: 42.5 (#194)

Multilingual benchmarks
Benchmark	Devstral Small 2505	Gemma 3 4B
LMArena Non-English	—	1273
LMArena German	—	1281
LMArena Russian	—	1294

Instruction Following Not comparable

Devstral Small 2505: —, Gemma 3 4B: 65.2 (#225)

Instruction Following benchmarks
Benchmark	Devstral Small 2505	Gemma 3 4B
LMArena Instruction Following	—	1239

Long Context Not comparable

Devstral Small 2505: —, Gemma 3 4B: 38.7 (#194)

Long Context benchmarks
Benchmark	Devstral Small 2505	Gemma 3 4B
LMArena Longer Query	—	1273

Writing & Preference Not comparable

Devstral Small 2505: —, Gemma 3 4B: 42.0 (#239)

Writing & Preference benchmarks
Benchmark	Devstral Small 2505	Gemma 3 4B
LMArena Text	—	1291
LMArena Creative Writing	—	1271
EQ-Bench Creative Writing	—	1068
LMArena Multi-Turn	—	1255

Frequently asked questions

Is Devstral Small 2505 better than Gemma 3 4B?

Which is cheaper, Devstral Small 2505 or Gemma 3 4B?

Gemma 3 4B is cheaper. It lists at $0.04 per million input tokens and $0.08 per million output tokens; Devstral Small 2505 lists at $0.10 and $0.30.

Is Devstral Small 2505 or Gemma 3 4B better for coding?

Devstral Small 2505 scores higher on coding benchmarks: 38.9 versus 35.9 in the Noometry coding category.

Which has the bigger context window?

Gemma 3 4B does, with 131K tokens against 128K.

How many benchmarks do Devstral Small 2505 and Gemma 3 4B share?

1 benchmark has published results for both models. Devstral Small 2505 has 4 scored results on Noometry and Gemma 3 4B has 22.