Noometry

Model comparison

Devstral Small 2505 vs Grok 4

Grok 4 is the stronger model overall, scoring 47.8 to 33.9 on the Noometry Index.

Last verified . 1 shared benchmarks.

Devstral Small 2505 Mistral AI

33.9

Rank #228 Reported

Grok 4 xAI

47.8

Rank #49 Confirmed

Summary

  • They share 1 benchmark with published results for both. Devstral Small 2505 scores higher in 0 categories and Grok 4 in 2 categories; 2 gaps are clear of the uncertainty.
  • The widest gap is in reasoning, where Grok 4 leads 36.9 to 19.1.
  • The biggest single-benchmark swing is Kagi LLM Benchmark: 37.7% for Devstral Small 2505 and 73.6% for Grok 4.
  • Devstral Small 2505 has downloadable open weights; the other is API-only.

Side by side

Devstral Small 2505Grok 4
ProviderMistral AIxAI
Noometry Index33.947.8
Released2025-05-072025-07-09
WeightsOpenProprietary
Context window128K—
Max output128K—
Input $ / M tokens$0.10—
Output $ / M tokens$0.30—
Results tracked448

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Grok 4 leads

Devstral Small 2505: 38.6 (#160), Grok 4: 49.9 (#44)

BenchmarkDevstral Small 2505Grok 4
SWE-bench Verified (bash only)56.4%—
Aider Polyglot—79.6%
SciCode28.8%—
WeirdML—45.7%
LMArena Coding—1408

Agentic & Tool Use Not comparable

Devstral Small 2505: —, Grok 4: 32.1 (#62)

BenchmarkDevstral Small 2505Grok 4
Terminal-Bench—27.2%
Berkeley Function Calling Leaderboard—63%
GDPval—21.1%
Cybench—43%
DeepResearch Bench—47.3%
BALROG—43.6%
LMArena Search—1142
METR Time Horizons—66.6%

Reasoning Grok 4 leads

Devstral Small 2505: 19.1 (#243), Grok 4: 36.9 (#59)

BenchmarkDevstral Small 2505Grok 4
Kagi LLM Benchmark37.7%73.6%
ARC-AGI-2—16%
SimpleBench—60.5%
ARC-AGI-1—66.7%
CritPt0%—
Chess Puzzles—28%
LMArena Hard Prompts—1409
Epoch Capabilities Index—146.46
ForecastBench—60.9

Math Not comparable

Devstral Small 2505: —, Grok 4: 46.5 (#58)

BenchmarkDevstral Small 2505Grok 4
OTIS Mock AIME 2024-2025—84%
Omni-MATH—60.3%
LMArena Math—1422
FrontierMath (Feb 2025 set)—19.7%
FrontierMath Tier 4 (v1)—2.1%

Knowledge Not comparable

Devstral Small 2505: —, Grok 4: 53.7 (#50)

BenchmarkDevstral Small 2505Grok 4
GPQA Diamond—87%
MMLU-Pro—85.1%
Confabulations—12.4%
GPQA (HELM)—72.7%
LMArena Expert—1417

Multimodal Not comparable

Devstral Small 2505: —, Grok 4: 33.1 (#89)

BenchmarkDevstral Small 2505Grok 4
LMArena Vision—1211
GeoBench—45%

Multilingual Not comparable

Devstral Small 2505: —, Grok 4: 52.0 (#98)

BenchmarkDevstral Small 2505Grok 4
LMArena Non-English—1403
LMArena Chinese—1428
LMArena French—1418
LMArena German—1429
LMArena Japanese—1396
LMArena Korean—1376
LMArena Russian—1409
LMArena Spanish—1419

Instruction Following Not comparable

Devstral Small 2505: —, Grok 4: 79.4 (#4)

BenchmarkDevstral Small 2505Grok 4
IFEval—94.9%
LMArena Instruction Following—1387

Long Context Not comparable

Devstral Small 2505: —, Grok 4: 63.1 (#4)

BenchmarkDevstral Small 2505Grok 4
Fiction.LiveBench—94.4%
LMArena Longer Query—1409

Writing & Preference Not comparable

Devstral Small 2505: —, Grok 4: 58.7 (#111)

BenchmarkDevstral Small 2505Grok 4
LMArena Text—1412
LMArena Creative Writing—1397
Short-Story Creative Writing—76.9%
WildBench—79.7%
LMArena Multi-Turn—1416

Frequently asked questions

Is Devstral Small 2505 better than Grok 4?

Grok 4 is the stronger model overall, scoring 47.8 to 33.9 on the Noometry Index.

Is Devstral Small 2505 or Grok 4 better for coding?

Grok 4 scores higher on coding benchmarks: 49.9 versus 38.6 in the Noometry coding category.

How many benchmarks do Devstral Small 2505 and Grok 4 share?

1 benchmark has published results for both models. Devstral Small 2505 has 4 scored results on Noometry and Grok 4 has 48.

Related comparisons

Go deeper