Noometry

Model comparison

Devstral Small 2505 vs Llama 3.1-70B

Devstral Small 2505 is the stronger model overall, scoring 34.3 to 29.6 on the Noometry Index.

Last verified . 0 shared benchmarks.

Devstral Small 2505 Mistral AI

34.3

Rank #233 Reported

Llama 3.1-70B Meta

29.6

Rank #308 Confirmed

Summary

  • The widest gap is in coding, where Devstral Small 2505 leads 38.9 to 30.3.
  • Devstral Small 2505 is cheaper at $0.10 / $0.30 per million input/output tokens, against $0.40 / $0.40 for Llama 3.1-70B.

Side by side

Devstral Small 2505Llama 3.1-70B
ProviderMistral AIMeta
Noometry Index34.329.6
Released2025-05-072024-07-23
WeightsOpenOpen
Context window128K128K
Max output128K4K
Input $ / M tokens$0.10$0.40
Output $ / M tokens$0.30$0.40
Results tracked435

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Devstral Small 2505 leads

Devstral Small 2505: 38.9 (#166), Llama 3.1-70B: 30.3 (#296)

BenchmarkDevstral Small 2505Llama 3.1-70B
SWE-bench Verified (bash only)56.4%—
SciCode28.8%—
WeirdML—9%
BigCodeBench Instruct—46.1%
LMArena Coding—1260
BigCodeBench Complete—54.8%

Agentic & Tool Use Not comparable

Devstral Small 2505: —, Llama 3.1-70B: 25.1 (#112)

BenchmarkDevstral Small 2505Llama 3.1-70B
TheAgentCompany—6.9%
BALROG—27.9%

Reasoning Llama 3.1-70B leads

Devstral Small 2505: 19.7 (#252), Llama 3.1-70B: 21.6 (#220)

BenchmarkDevstral Small 2505Llama 3.1-70B
Kagi LLM Benchmark37.7%—
CritPt0%—
LMArena Hard Prompts—1241
DTBench—60%
LMCA—14.8%
Epoch Capabilities Index—125.92

Math Not comparable

Devstral Small 2505: —, Llama 3.1-70B: 13.5 (#304)

BenchmarkDevstral Small 2505Llama 3.1-70B
OTIS Mock AIME 2024-2025—3.6%
Omni-MATH—21%
LMArena Math—1252
MATH Level 5—36.7%

Knowledge Not comparable

Devstral Small 2505: —, Llama 3.1-70B: 24.2 (#269)

BenchmarkDevstral Small 2505Llama 3.1-70B
GPQA Diamond—44.2%
MMLU-Pro—65.3%
GPQA (HELM)—42.6%
LMArena Expert—1209
MMLU—80.1%

Multilingual Not comparable

Devstral Small 2505: —, Llama 3.1-70B: 38.8 (#225)

BenchmarkDevstral Small 2505Llama 3.1-70B
LMArena Non-English—1219
LMArena Chinese—1215
LMArena French—1261
LMArena German—1222
LMArena Japanese—1132
LMArena Korean—1140
LMArena Russian—1234
LMArena Spanish—1253

Instruction Following Not comparable

Devstral Small 2505: —, Llama 3.1-70B: 65.3 (#223)

BenchmarkDevstral Small 2505Llama 3.1-70B
IFEval—82.1%
LMArena Instruction Following—1231

Long Context Not comparable

Devstral Small 2505: —, Llama 3.1-70B: 37.6 (#214)

BenchmarkDevstral Small 2505Llama 3.1-70B
LMArena Longer Query—1241

Writing & Preference Not comparable

Devstral Small 2505: —, Llama 3.1-70B: 35.4 (#267)

BenchmarkDevstral Small 2505Llama 3.1-70B
LMArena Text—1261
LMArena Creative Writing—1232
EQ-Bench Creative Writing—784
WildBench—75.8%
LMArena Multi-Turn—1256

Frequently asked questions

Is Devstral Small 2505 better than Llama 3.1-70B?

Devstral Small 2505 is the stronger model overall, scoring 34.3 to 29.6 on the Noometry Index.

Which is cheaper, Devstral Small 2505 or Llama 3.1-70B?

Devstral Small 2505 is cheaper. It lists at $0.10 per million input tokens and $0.30 per million output tokens; Llama 3.1-70B lists at $0.40 and $0.40.

Is Devstral Small 2505 or Llama 3.1-70B better for coding?

Devstral Small 2505 scores higher on coding benchmarks: 38.9 versus 30.3 in the Noometry coding category.

Which has the bigger context window?

Both accept 128K tokens.

How many benchmarks do Devstral Small 2505 and Llama 3.1-70B share?

0 benchmarks have published results for both models. Devstral Small 2505 has 4 scored results on Noometry and Llama 3.1-70B has 35.

Related comparisons

Go deeper