Noometry

Model comparison

Devstral Small 2505 vs GPT-4

Devstral Small 2505 is the stronger model overall, scoring 34.3 to 29.1 on the Noometry Index.

Last verified . 0 shared benchmarks.

Devstral Small 2505 Mistral AI

34.3

Rank #233 Reported

GPT-4 OpenAI

29.1

Rank #316 Confirmed

Summary

  • The widest gap is in coding, where Devstral Small 2505 leads 38.9 to 31.6.
  • Devstral Small 2505 is cheaper at $0.10 / $0.30 per million input/output tokens, against $30 / $60 for GPT-4.
  • Devstral Small 2505 accepts more context: 128K tokens versus 8K.
  • Devstral Small 2505 has downloadable open weights; the other is API-only.

Side by side

Devstral Small 2505GPT-4
ProviderMistral AIOpenAI
Noometry Index34.329.1
Released2025-05-072023-03-14
WeightsOpenProprietary
Context window128K8K
Max output128K8K
Input $ / M tokens$0.10$30
Output $ / M tokens$0.30$60
Results tracked438

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Devstral Small 2505 leads

Devstral Small 2505: 38.9 (#166), GPT-4: 31.6 (#283)

BenchmarkDevstral Small 2505GPT-4
SWE-bench Verified (bash only)56.4%—
SciCode28.8%—
WeirdML—12.4%
BigCodeBench Instruct—46%
LMArena Coding—1254
BigCodeBench Complete—57.2%
HumanEval+—79.3%

Agentic & Tool Use Not comparable

Devstral Small 2505: —, GPT-4: —

BenchmarkDevstral Small 2505GPT-4
METR Time Horizons—36.1%

Reasoning Devstral Small 2505 leads

Devstral Small 2505: 19.7 (#252), GPT-4: 17.8 (#289)

BenchmarkDevstral Small 2505GPT-4
Kagi LLM Benchmark37.7%—
CritPt0%—
Chess Puzzles—4%
LMArena Hard Prompts—1241
Mystery Game Puzzles—12%
DTBench—62.7%
LMCA—17.1%
BIG-Bench Hard—75.1%
Epoch Capabilities Index—125.89
ForecastBench—57.8
HellaSwag—95.3%
WinoGrande—87.5%

Math Not comparable

Devstral Small 2505: —, GPT-4: 10.8 (#309)

BenchmarkDevstral Small 2505GPT-4
OTIS Mock AIME 2024-2025—1.1%
LMArena Math—1269
MATH Level 5—23%
GSM8K—92%

Knowledge Not comparable

Devstral Small 2505: —, GPT-4: 18.4 (#282)

BenchmarkDevstral Small 2505GPT-4
GPQA Diamond—35.7%
LMArena Expert—1211
MMLU—86.4%
TriviaQA—84.8%

Multilingual Not comparable

Devstral Small 2505: —, GPT-4: 40.6 (#215)

BenchmarkDevstral Small 2505GPT-4
LMArena Non-English—1246
LMArena Chinese—1242
LMArena French—1283
LMArena German—1251
LMArena Japanese—1209
LMArena Korean—1184
LMArena Russian—1251
LMArena Spanish—1261

Instruction Following Not comparable

Devstral Small 2505: —, GPT-4: 65.3 (#222)

BenchmarkDevstral Small 2505GPT-4
LMArena Instruction Following—1241

Long Context Not comparable

Devstral Small 2505: —, GPT-4: 37.7 (#212)

BenchmarkDevstral Small 2505GPT-4
LMArena Longer Query—1244

Writing & Preference Not comparable

Devstral Small 2505: —, GPT-4: 34.9 (#268)

BenchmarkDevstral Small 2505GPT-4
LMArena Text—1263
LMArena Creative Writing—1244
EQ-Bench Creative Writing—752
LMArena Multi-Turn—1257

Frequently asked questions

Is Devstral Small 2505 better than GPT-4?

Devstral Small 2505 is the stronger model overall, scoring 34.3 to 29.1 on the Noometry Index.

Which is cheaper, Devstral Small 2505 or GPT-4?

Devstral Small 2505 is cheaper. It lists at $0.10 per million input tokens and $0.30 per million output tokens; GPT-4 lists at $30 and $60.

Is Devstral Small 2505 or GPT-4 better for coding?

Devstral Small 2505 scores higher on coding benchmarks: 38.9 versus 31.6 in the Noometry coding category.

Which has the bigger context window?

Devstral Small 2505 does, with 128K tokens against 8K.

How many benchmarks do Devstral Small 2505 and GPT-4 share?

0 benchmarks have published results for both models. Devstral Small 2505 has 4 scored results on Noometry and GPT-4 has 38.

Related comparisons

Go deeper