Noometry

Model comparison

Devstral Small 2505 vs GPT-3.5-turbo

Devstral Small 2505 is the stronger model overall, scoring 34.3 to 23.2 on the Noometry Index.

Last verified . 0 shared benchmarks.

Devstral Small 2505 Mistral AI

34.3

Rank #233 Reported

GPT-3.5-turbo OpenAI

23.2

Rank #350 Confirmed

Summary

  • The widest gap is in coding, where Devstral Small 2505 leads 38.9 to 23.9.
  • Devstral Small 2505 is cheaper at $0.10 / $0.30 per million input/output tokens, against $0.50 / $1.50 for GPT-3.5-turbo.
  • Devstral Small 2505 accepts more context: 128K tokens versus 16K.
  • Devstral Small 2505 has downloadable open weights; the other is API-only.

Side by side

Devstral Small 2505GPT-3.5-turbo
ProviderMistral AIOpenAI
Noometry Index34.323.2
Released2025-05-072023-03-01
WeightsOpenProprietary
Context window128K16K
Max output128K4K
Input $ / M tokens$0.10$0.50
Output $ / M tokens$0.30$1.50
Results tracked444

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Devstral Small 2505 leads

Devstral Small 2505: 38.9 (#166), GPT-3.5-turbo: 23.9 (#331)

BenchmarkDevstral Small 2505GPT-3.5-turbo
SWE-bench Verified (bash only)56.4%—
SciCode28.8%—
WeirdML—3.5%
BigCodeBench Instruct—39.1%
LMArena Coding—1136
BigCodeBench Complete—50.6%
HumanEval+—70.7%
MBPP+—69.7%

Agentic & Tool Use Not comparable

Devstral Small 2505: —, GPT-3.5-turbo: —

BenchmarkDevstral Small 2505GPT-3.5-turbo
METR Time Horizons—21.5%

Reasoning Devstral Small 2505 leads

Devstral Small 2505: 19.7 (#252), GPT-3.5-turbo: 13.8 (#332)

BenchmarkDevstral Small 2505GPT-3.5-turbo
Kagi LLM Benchmark37.7%—
CritPt0%—
Chess Puzzles—0%
LMArena Hard Prompts—1108
Mystery Game Puzzles—3%
DTBench—48.5%
LMCA—9.7%
Adversarial NLI—58.1%
BIG-Bench Hard—61.6%
CommonsenseQA 2.0—57%
Epoch Capabilities Index—118.55
ForecastBench—50.4
WinoGrande—81.6%

Math Not comparable

Devstral Small 2505: —, GPT-3.5-turbo: 6.3 (#327)

BenchmarkDevstral Small 2505GPT-3.5-turbo
FrontierMath (Tiers 1-3)—0%
OTIS Mock AIME 2024-2025—2.2%
LMArena Math—1142
MATH Level 5—15.9%
GSM8K—57.8%

Knowledge Not comparable

Devstral Small 2505: —, GPT-3.5-turbo: 10.0 (#303)

BenchmarkDevstral Small 2505GPT-3.5-turbo
GPQA Diamond—28%
LMArena Expert—1070
ARC (AI2) Challenge—87.4%
BoolQ—87%
MMLU—71.4%
OpenBookQA—86%
TriviaQA—85.8%

Multilingual Not comparable

Devstral Small 2505: —, GPT-3.5-turbo: 31.5 (#258)

BenchmarkDevstral Small 2505GPT-3.5-turbo
LMArena Non-English—1108
LMArena Chinese—1075
LMArena French—1118
LMArena German—1090
LMArena Japanese—1043
LMArena Korean—1019
LMArena Russian—1123
LMArena Spanish—1121

Instruction Following Not comparable

Devstral Small 2505: —, GPT-3.5-turbo: 57.9 (#262)

BenchmarkDevstral Small 2505GPT-3.5-turbo
LMArena Instruction Following—1119

Long Context Not comparable

Devstral Small 2505: —, GPT-3.5-turbo: 34.0 (#254)

BenchmarkDevstral Small 2505GPT-3.5-turbo
LMArena Longer Query—1121

Writing & Preference Not comparable

Devstral Small 2505: —, GPT-3.5-turbo: 25.3 (#305)

BenchmarkDevstral Small 2505GPT-3.5-turbo
LMArena Text—1125
LMArena Creative Writing—1092
EQ-Bench Creative Writing—451
LMArena Multi-Turn—1117

Frequently asked questions

Is Devstral Small 2505 better than GPT-3.5-turbo?

Devstral Small 2505 is the stronger model overall, scoring 34.3 to 23.2 on the Noometry Index.

Which is cheaper, Devstral Small 2505 or GPT-3.5-turbo?

Devstral Small 2505 is cheaper. It lists at $0.10 per million input tokens and $0.30 per million output tokens; GPT-3.5-turbo lists at $0.50 and $1.50.

Is Devstral Small 2505 or GPT-3.5-turbo better for coding?

Devstral Small 2505 scores higher on coding benchmarks: 38.9 versus 23.9 in the Noometry coding category.

Which has the bigger context window?

Devstral Small 2505 does, with 128K tokens against 16K.

How many benchmarks do Devstral Small 2505 and GPT-3.5-turbo share?

0 benchmarks have published results for both models. Devstral Small 2505 has 4 scored results on Noometry and GPT-3.5-turbo has 44.

Related comparisons

Go deeper