Noometry

Model comparison

Mistral Nemo vs Qwen1.5-32B

Qwen1.5-32B is the stronger model overall, scoring 30.5 to 26.4 on the Noometry Index.

Last verified . 1 shared benchmarks.

Mistral Nemo Mistral AI

26.4

Rank #337 Confirmed

Qwen1.5-32B Alibaba (Qwen)

30.5

Rank #293 Confirmed

Summary

  • They share 1 benchmark with published results for both. Mistral Nemo scores higher in 0 categories and Qwen1.5-32B in 4 categories; 4 gaps are clear of the uncertainty.
  • The widest gap is in math, where Qwen1.5-32B leads 33.0 to 25.5.

Side by side

Mistral NemoQwen1.5-32B
ProviderMistral AIAlibaba (Qwen)
Noometry Index26.430.5
Released2024-07-012024-02-04
WeightsOpenOpen
Context window128K—
Max output128K—
Input $ / M tokens$0.15—
Output $ / M tokens$0.15—
Results tracked1021

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Mistral Nemo: —, Qwen1.5-32B: 31.7 (#282)

BenchmarkMistral NemoQwen1.5-32B
BigCodeBench Instruct—32.3%
LMArena Coding—1155
BigCodeBench Complete—42%

Agentic & Tool Use Not comparable

Mistral Nemo: 23.5 (#125), Qwen1.5-32B: —

BenchmarkMistral NemoQwen1.5-32B
Berkeley Function Calling Leaderboard27.6%—
BALROG17.6%—

Reasoning Qwen1.5-32B leads

Mistral Nemo: 20.7 (#232), Qwen1.5-32B: 21.8 (#212)

BenchmarkMistral NemoQwen1.5-32B
LMArena Hard Prompts—1130
DTBench48.6%—
Epoch Capabilities Index118.68—
PIQA83.5%—

Math Qwen1.5-32B leads

Mistral Nemo: 25.5 (#268), Qwen1.5-32B: 33.0 (#207)

BenchmarkMistral NemoQwen1.5-32B
LMArena Math—1155
MATH Level 510.8%—
GSM8K84.2%—

Knowledge Qwen1.5-32B leads

Mistral Nemo: 12.3 (#298), Qwen1.5-32B: 13.5 (#296)

BenchmarkMistral NemoQwen1.5-32B
GPQA Diamond29.9%30.7%
LMArena Expert—1126
BoolQ82.5%—
MMLU—74.4%

Multilingual Not comparable

Mistral Nemo: —, Qwen1.5-32B: 31.4 (#259)

BenchmarkMistral NemoQwen1.5-32B
LMArena Non-English—1106
LMArena Chinese—1177
LMArena French—1101
LMArena German—1058
LMArena Japanese—1027
LMArena Korean—1008
LMArena Russian—1073
LMArena Spanish—1089

Instruction Following Not comparable

Mistral Nemo: —, Qwen1.5-32B: 57.7 (#265)

BenchmarkMistral NemoQwen1.5-32B
LMArena Instruction Following—1116

Long Context Not comparable

Mistral Nemo: —, Qwen1.5-32B: 34.7 (#246)

BenchmarkMistral NemoQwen1.5-32B
LMArena Longer Query—1146

Writing & Preference Qwen1.5-32B leads

Mistral Nemo: 28.5 (#296), Qwen1.5-32B: 34.2 (#271)

BenchmarkMistral NemoQwen1.5-32B
LMArena Text—1137
LMArena Creative Writing—1083
EQ-Bench Creative Writing881—
LMArena Multi-Turn—1140

Frequently asked questions

Is Mistral Nemo better than Qwen1.5-32B?

Qwen1.5-32B is the stronger model overall, scoring 30.5 to 26.4 on the Noometry Index.

How many benchmarks do Mistral Nemo and Qwen1.5-32B share?

1 benchmark has published results for both models. Mistral Nemo has 10 scored results on Noometry and Qwen1.5-32B has 21.

Related comparisons

Go deeper