Noometry

Model comparison

DeepSeek LLM 67B vs Grok-2 (Dec 2024)

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.7 to 24.9 on the Noometry Index.

Last verified . 14 shared benchmarks.

DeepSeek LLM 67B DeepSeek

24.9

Rank #347 Confirmed

Grok-2 (Dec 2024) xAI

33.7

Rank #239 Confirmed

Summary

  • They share 14 benchmarks with published results for both. DeepSeek LLM 67B scores higher in 0 categories and Grok-2 (Dec 2024) in 8 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Grok-2 (Dec 2024) leads 29.8 to 7.0.
  • The biggest single-benchmark swing is MATH Level 5: 6.4% for DeepSeek LLM 67B and 63.5% for Grok-2 (Dec 2024).
  • DeepSeek LLM 67B has downloadable open weights; the other is API-only.

Side by side

DeepSeek LLM 67BGrok-2 (Dec 2024)
ProviderDeepSeekxAI
Noometry Index24.933.7
Released2023-11-292024-08-13
WeightsOpenProprietary
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked1534

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Grok-2 (Dec 2024) leads

DeepSeek LLM 67B: 31.9 (#278), Grok-2 (Dec 2024): 33.3 (#258)

BenchmarkDeepSeek LLM 67BGrok-2 (Dec 2024)
LMArena Coding10961287
WeirdML—22.2%
LiveBench Coding—46.4%

Reasoning Too close to call

DeepSeek LLM 67B: 16.5 (#304), Grok-2 (Dec 2024): 16.9 (#299)

BenchmarkDeepSeek LLM 67BGrok-2 (Dec 2024)
LMArena Hard Prompts10701272
Epoch Capabilities Index110.5130.48
SimpleBench—22.7%
Chess Puzzles0%—
LiveBench Reasoning—54.8%
DTBench—65.2%
LiveBench Data Analysis—54.5%
LiveBench—54.3%

Math Grok-2 (Dec 2024) leads

DeepSeek LLM 67B: 8.7 (#324), Grok-2 (Dec 2024): 20.8 (#284)

BenchmarkDeepSeek LLM 67BGrok-2 (Dec 2024)
OTIS Mock AIME 2024-20250.8%11.5%
LMArena Math11081283
MATH Level 56.4%63.5%
LiveBench Math—54.9%
FrontierMath (Feb 2025 set)—0.7%

Knowledge Grok-2 (Dec 2024) leads

DeepSeek LLM 67B: 7.0 (#313), Grok-2 (Dec 2024): 29.8 (#233)

BenchmarkDeepSeek LLM 67BGrok-2 (Dec 2024)
GPQA Diamond24.6%53.8%
Confabulations—20.1%
LMArena Expert—1254

Multilingual Grok-2 (Dec 2024) leads

DeepSeek LLM 67B: 29.4 (#267), Grok-2 (Dec 2024): 43.1 (#188)

BenchmarkDeepSeek LLM 67BGrok-2 (Dec 2024)
LMArena Non-English10731282
LMArena Chinese11321289
LMArena French—1318
LMArena German—1287
LMArena Japanese—1244
LMArena Korean—1237
LMArena Russian—1286
LMArena Spanish—1281

Instruction Following Grok-2 (Dec 2024) leads

DeepSeek LLM 67B: 55.4 (#277), Grok-2 (Dec 2024): 66.9 (#202)

BenchmarkDeepSeek LLM 67BGrok-2 (Dec 2024)
LMArena Instruction Following10791270
LiveBench Instruction Following—69.6%

Long Context Grok-2 (Dec 2024) leads

DeepSeek LLM 67B: 33.1 (#265), Grok-2 (Dec 2024): 38.8 (#190)

BenchmarkDeepSeek LLM 67BGrok-2 (Dec 2024)
LMArena Longer Query10921276

Writing & Preference Grok-2 (Dec 2024) leads

DeepSeek LLM 67B: 31.6 (#282), Grok-2 (Dec 2024): 48.6 (#198)

BenchmarkDeepSeek LLM 67BGrok-2 (Dec 2024)
LMArena Text11051305
LMArena Creative Writing10671284
LMArena Multi-Turn10821290
Short-Story Creative Writing—63.6%
LiveBench Language—45.6%

Frequently asked questions

Is DeepSeek LLM 67B better than Grok-2 (Dec 2024)?

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.7 to 24.9 on the Noometry Index.

Is DeepSeek LLM 67B or Grok-2 (Dec 2024) better for coding?

Grok-2 (Dec 2024) scores higher on coding benchmarks: 33.3 versus 31.9 in the Noometry coding category.

How many benchmarks do DeepSeek LLM 67B and Grok-2 (Dec 2024) share?

14 benchmarks have published results for both models. DeepSeek LLM 67B has 15 scored results on Noometry and Grok-2 (Dec 2024) has 34.

Related comparisons

Go deeper