Noometry

Model comparison

Grok-2 (Dec 2024) vs Mistral Nemo

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.7 to 26.4 on the Noometry Index.

Last verified . 4 shared benchmarks.

Grok-2 (Dec 2024) xAI

33.7

Rank #239 Confirmed

Mistral Nemo Mistral AI

26.4

Rank #337 Confirmed

Summary

  • They share 4 benchmarks with published results for both. Grok-2 (Dec 2024) scores higher in 2 categories and Mistral Nemo in 2 categories; 4 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Grok-2 (Dec 2024) leads 48.6 to 28.5.
  • The biggest single-benchmark swing is MATH Level 5: 63.5% for Grok-2 (Dec 2024) and 10.8% for Mistral Nemo.
  • Mistral Nemo has downloadable open weights; the other is API-only.

Side by side

Grok-2 (Dec 2024)Mistral Nemo
ProviderxAIMistral AI
Noometry Index33.726.4
Released2024-08-132024-07-01
WeightsProprietaryOpen
Context window—128K
Max output—128K
Input $ / M tokens—$0.15
Output $ / M tokens—$0.15
Results tracked3410

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Grok-2 (Dec 2024): 33.3 (#258), Mistral Nemo: —

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
WeirdML22.2%—
LiveBench Coding46.4%—
LMArena Coding1287—

Agentic & Tool Use Not comparable

Grok-2 (Dec 2024): —, Mistral Nemo: 23.5 (#125)

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
Berkeley Function Calling Leaderboard—27.6%
BALROG—17.6%

Reasoning Mistral Nemo leads

Grok-2 (Dec 2024): 16.9 (#299), Mistral Nemo: 20.7 (#232)

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
DTBench65.2%48.6%
Epoch Capabilities Index130.48118.68
SimpleBench22.7%—
LiveBench Reasoning54.8%—
LMArena Hard Prompts1272—
LiveBench Data Analysis54.5%—
LiveBench54.3%—
PIQA—83.5%

Math Mistral Nemo leads

Grok-2 (Dec 2024): 20.8 (#284), Mistral Nemo: 25.5 (#268)

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
MATH Level 563.5%10.8%
OTIS Mock AIME 2024-202511.5%—
LiveBench Math54.9%—
LMArena Math1283—
FrontierMath (Feb 2025 set)0.7%—
GSM8K—84.2%

Knowledge Grok-2 (Dec 2024) leads

Grok-2 (Dec 2024): 29.8 (#233), Mistral Nemo: 12.3 (#298)

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
GPQA Diamond53.8%29.9%
Confabulations20.1%—
LMArena Expert1254—
BoolQ—82.5%

Multilingual Not comparable

Grok-2 (Dec 2024): 43.1 (#188), Mistral Nemo: —

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
LMArena Non-English1282—
LMArena Chinese1289—
LMArena French1318—
LMArena German1287—
LMArena Japanese1244—
LMArena Korean1237—
LMArena Russian1286—
LMArena Spanish1281—

Instruction Following Not comparable

Grok-2 (Dec 2024): 66.9 (#202), Mistral Nemo: —

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
LiveBench Instruction Following69.6%—
LMArena Instruction Following1270—

Long Context Not comparable

Grok-2 (Dec 2024): 38.8 (#190), Mistral Nemo: —

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
LMArena Longer Query1276—

Writing & Preference Grok-2 (Dec 2024) leads

Grok-2 (Dec 2024): 48.6 (#198), Mistral Nemo: 28.5 (#296)

BenchmarkGrok-2 (Dec 2024)Mistral Nemo
LMArena Text1305—
LMArena Creative Writing1284—
Short-Story Creative Writing63.6%—
EQ-Bench Creative Writing—881
LMArena Multi-Turn1290—
LiveBench Language45.6%—

Frequently asked questions

Is Grok-2 (Dec 2024) better than Mistral Nemo?

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.7 to 26.4 on the Noometry Index.

How many benchmarks do Grok-2 (Dec 2024) and Mistral Nemo share?

4 benchmarks have published results for both models. Grok-2 (Dec 2024) has 34 scored results on Noometry and Mistral Nemo has 10.

Related comparisons

Go deeper