Noometry

Model comparison

Claude 3 Haiku vs DeepSeek LLM 67B

Claude 3 Haiku and DeepSeek LLM 67B score almost the same on the Noometry Index (25.7 vs 24.8), so choose on price, context window or the category you care about most.

Last verified . 14 shared benchmarks.

Claude 3 Haiku Anthropic

25.7

Rank #333 Confirmed

DeepSeek LLM 67B DeepSeek

24.8

Rank #341 Confirmed

Summary

  • They share 14 benchmarks with published results for both. Claude 3 Haiku scores higher in 5 categories and DeepSeek LLM 67B in 3 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Claude 3 Haiku leads 17.3 to 7.1.
  • The biggest single-benchmark swing is GPQA Diamond: 36.3% for Claude 3 Haiku and 24.6% for DeepSeek LLM 67B.
  • DeepSeek LLM 67B has downloadable open weights; the other is API-only.

Side by side

Claude 3 HaikuDeepSeek LLM 67B
ProviderAnthropicDeepSeek
Noometry Index25.724.8
Released2024-03-072023-11-29
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked3715

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding DeepSeek LLM 67B leads

Claude 3 Haiku: 26.1 (#320), DeepSeek LLM 67B: 31.3 (#273)

BenchmarkClaude 3 HaikuDeepSeek LLM 67B
LMArena Coding11991096
WeirdML9.8%—
BigCodeBench Instruct39.4%—
BigCodeBench Complete50.1%—
CadEval12%—
HumanEval+68.9%—
MBPP+68.8%—

Reasoning Too close to call

Claude 3 Haiku: 15.7 (#303), DeepSeek LLM 67B: 15.9 (#300)

BenchmarkClaude 3 HaikuDeepSeek LLM 67B
LMArena Hard Prompts11741070
Epoch Capabilities Index118.3110.54
Kagi LLM Benchmark34.2%—
Chess Puzzles—0%
DTBench50.1%—
LMCA8.8%—
ForecastBench53.2—
WinoGrande74.2%—

Math Claude 3 Haiku leads

Claude 3 Haiku: 9.2 (#313), DeepSeek LLM 67B: 8.1 (#318)

BenchmarkClaude 3 HaikuDeepSeek LLM 67B
OTIS Mock AIME 2024-20251.8%0.8%
LMArena Math11881108
MATH Level 514.9%6.4%

Knowledge Claude 3 Haiku leads

Claude 3 Haiku: 17.3 (#280), DeepSeek LLM 67B: 7.1 (#308)

BenchmarkClaude 3 HaikuDeepSeek LLM 67B
GPQA Diamond36.3%24.6%
Confabulations34.2%—
LMArena Expert1148—
MMLU73.8%—

Multimodal Not comparable

Claude 3 Haiku: 23.1 (#123), DeepSeek LLM 67B: —

BenchmarkClaude 3 HaikuDeepSeek LLM 67B
LMArena Vision950—
ScienceQA72%—

Multilingual Claude 3 Haiku leads

Claude 3 Haiku: 36.1 (#238), DeepSeek LLM 67B: 29.5 (#262)

BenchmarkClaude 3 HaikuDeepSeek LLM 67B
LMArena Non-English11781073
LMArena Chinese11551132
LMArena French1194—
LMArena German1174—
LMArena Japanese1102—
LMArena Korean1108—
LMArena Russian1204—
LMArena Spanish1165—

Instruction Following Claude 3 Haiku leads

Claude 3 Haiku: 61.5 (#242), DeepSeek LLM 67B: 55.7 (#272)

BenchmarkClaude 3 HaikuDeepSeek LLM 67B
LMArena Instruction Following11741079

Long Context Claude 3 Haiku leads

Claude 3 Haiku: 36.1 (#232), DeepSeek LLM 67B: 33.1 (#260)

BenchmarkClaude 3 HaikuDeepSeek LLM 67B
LMArena Longer Query11911092

Writing & Preference DeepSeek LLM 67B leads

Claude 3 Haiku: 29.8 (#286), DeepSeek LLM 67B: 31.8 (#277)

BenchmarkClaude 3 HaikuDeepSeek LLM 67B
LMArena Text11951105
LMArena Creative Writing11571067
LMArena Multi-Turn11901082
EQ-Bench Creative Writing717—

Frequently asked questions

Is Claude 3 Haiku better than DeepSeek LLM 67B?

Claude 3 Haiku and DeepSeek LLM 67B score almost the same on the Noometry Index (25.7 vs 24.8), so choose on price, context window or the category you care about most.

Is Claude 3 Haiku or DeepSeek LLM 67B better for coding?

DeepSeek LLM 67B scores higher on coding benchmarks: 31.3 versus 26.1 in the Noometry coding category.

How many benchmarks do Claude 3 Haiku and DeepSeek LLM 67B share?

14 benchmarks have published results for both models. Claude 3 Haiku has 37 scored results on Noometry and DeepSeek LLM 67B has 15.

Related comparisons

Go deeper