Model comparison

Devstral Small 2505 vs Gemma 3 12B

Devstral Small 2505 is the stronger model overall, scoring 33.9 to 31.8 on the Noometry Index. Gemma 3 12B costs 2.0× less per token, which makes it the better buy when Devstral Small 2505's lead doesn't matter for your workload.

Last verified October 10, 2026. 2 shared benchmarks.

Devstral Small 2505

Mistral AI

33.9

Rank #228 Reported

Gemma 3 12B

Google

31.8

Rank #255 Confirmed

Summary

They share 2 benchmarks with published results for both. Devstral Small 2505 scores higher in 2 categories and Gemma 3 12B in 0 categories; 2 gaps are clear of the uncertainty.
The widest gap is in coding, where Devstral Small 2505 leads 38.6 to 31.2.
The biggest single-benchmark swing is SciCode: 28.8% for Devstral Small 2505 and 17.4% for Gemma 3 12B.
Gemma 3 12B is cheaper at $0.05 / $0.15 per million input/output tokens, against $0.10 / $0.30 for Devstral Small 2505.
Gemma 3 12B accepts more context: 131K tokens versus 128K.

Side by side

Devstral Small 2505 and Gemma 3 12B specifications
	Devstral Small 2505	Gemma 3 12B
Provider	Mistral AI	Google
Noometry Index	33.9	31.8
Released	2025-05-07	2025-03-12
Weights	Open	Open
Context window	128K	131K
Max output	128K	8K
Input $ / M tokens	$0.10	$0.05
Output $ / M tokens	$0.30	$0.15
Results tracked	4	24

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Devstral Small 2505 leads

Devstral Small 2505: 38.6 (#160), Gemma 3 12B: 31.2 (#276)

Coding benchmarks
Benchmark	Devstral Small 2505	Gemma 3 12B
SciCode	28.8%	17.4%
SWE-bench Verified (bash only)	56.4%	—
LMArena Coding	—	1281

Agentic & Tool Use Not comparable

Devstral Small 2505: —, Gemma 3 12B: 25.3 (#101)

Agentic & Tool Use benchmarks
Benchmark	Devstral Small 2505	Gemma 3 12B
Berkeley Function Calling Leaderboard	—	30.4%

Reasoning Devstral Small 2505 leads

Devstral Small 2505: 19.1 (#243), Gemma 3 12B: 15.2 (#307)

Reasoning benchmarks
Benchmark	Devstral Small 2505	Gemma 3 12B
CritPt	0%	0%
Kagi LLM Benchmark	37.7%	—
Chess Puzzles	—	0%
LMArena Hard Prompts	—	1309
DTBench	—	48.8%
LMCA	—	4.5%
Epoch Capabilities Index	—	123.49

Math Not comparable

Devstral Small 2505: —, Gemma 3 12B: 21.0 (#273)

Math benchmarks
Benchmark	Devstral Small 2505	Gemma 3 12B
OTIS Mock AIME 2024-2025	—	16.7%
LMArena Math	—	1307

Knowledge Not comparable

Devstral Small 2505: —, Gemma 3 12B: 26.5 (#250)

Knowledge benchmarks
Benchmark	Devstral Small 2505	Gemma 3 12B
GPQA Diamond	—	39.5%
Vectara Hallucination Rate	—	4.4%
LMArena Expert	—	1248

Multimodal Not comparable

Devstral Small 2505: —, Gemma 3 12B: —

Multimodal benchmarks
Benchmark	Devstral Small 2505	Gemma 3 12B
MindCube	—	46.7%

Multilingual Not comparable

Devstral Small 2505: —, Gemma 3 12B: 45.9 (#160)

Multilingual benchmarks
Benchmark	Devstral Small 2505	Gemma 3 12B
LMArena Non-English	—	1318
LMArena German	—	1370
LMArena Russian	—	1335

Instruction Following Not comparable

Devstral Small 2505: —, Gemma 3 12B: 68.8 (#179)

Instruction Following benchmarks
Benchmark	Devstral Small 2505	Gemma 3 12B
LMArena Instruction Following	—	1299

Long Context Not comparable

Devstral Small 2505: —, Gemma 3 12B: 40.0 (#157)

Long Context benchmarks
Benchmark	Devstral Small 2505	Gemma 3 12B
LMArena Longer Query	—	1317

Writing & Preference Not comparable

Devstral Small 2505: —, Gemma 3 12B: 47.7 (#205)

Writing & Preference benchmarks
Benchmark	Devstral Small 2505	Gemma 3 12B
LMArena Text	—	1334
LMArena Creative Writing	—	1331
EQ-Bench Creative Writing	—	1126
LMArena Multi-Turn	—	1334

Frequently asked questions

Is Devstral Small 2505 better than Gemma 3 12B?

Which is cheaper, Devstral Small 2505 or Gemma 3 12B?

Gemma 3 12B is cheaper. It lists at $0.05 per million input tokens and $0.15 per million output tokens; Devstral Small 2505 lists at $0.10 and $0.30.

Is Devstral Small 2505 or Gemma 3 12B better for coding?

Devstral Small 2505 scores higher on coding benchmarks: 38.6 versus 31.2 in the Noometry coding category.

Which has the bigger context window?

Gemma 3 12B does, with 131K tokens against 128K.

How many benchmarks do Devstral Small 2505 and Gemma 3 12B share?

2 benchmarks have published results for both models. Devstral Small 2505 has 4 scored results on Noometry and Gemma 3 12B has 24.