Noometry

Model comparison

Claude 3 Haiku vs Llama 3.1-70B

Llama 3.1-70B is the stronger model overall, scoring 29.6 to 25.9 on the Noometry Index.

Last verified . 28 shared benchmarks.

Claude 3 Haiku Anthropic

25.9

Rank #340 Confirmed

Llama 3.1-70B Meta

29.6

Rank #308 Confirmed

Summary

  • They share 28 benchmarks with published results for both. Claude 3 Haiku scores higher in 0 categories and Llama 3.1-70B in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Llama 3.1-70B leads 24.2 to 17.3.
  • The biggest single-benchmark swing is MATH Level 5: 14.9% for Claude 3 Haiku and 36.7% for Llama 3.1-70B.
  • Llama 3.1-70B has downloadable open weights; the other is API-only.

Side by side

Claude 3 HaikuLlama 3.1-70B
ProviderAnthropicMeta
Noometry Index25.929.6
Released2024-03-072024-07-23
WeightsProprietaryOpen
Context window—128K
Max output—4K
Input $ / M tokens—$0.40
Output $ / M tokens—$0.40
Results tracked3735

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Llama 3.1-70B leads

Claude 3 Haiku: 26.4 (#325), Llama 3.1-70B: 30.3 (#296)

BenchmarkClaude 3 HaikuLlama 3.1-70B
WeirdML9.8%9%
BigCodeBench Instruct39.4%46.1%
LMArena Coding11991260
BigCodeBench Complete50.1%54.8%
CadEval12%—
HumanEval+68.9%—
MBPP+68.8%—

Agentic & Tool Use Not comparable

Claude 3 Haiku: —, Llama 3.1-70B: 25.1 (#112)

BenchmarkClaude 3 HaikuLlama 3.1-70B
TheAgentCompany—6.9%
BALROG—27.9%

Reasoning Llama 3.1-70B leads

Claude 3 Haiku: 16.3 (#307), Llama 3.1-70B: 21.6 (#220)

BenchmarkClaude 3 HaikuLlama 3.1-70B
LMArena Hard Prompts11741241
DTBench50.1%60%
LMCA8.8%14.8%
Epoch Capabilities Index118.35125.92
Kagi LLM Benchmark34.2%—
ForecastBench53.2—
WinoGrande74.2%—

Math Llama 3.1-70B leads

Claude 3 Haiku: 9.8 (#319), Llama 3.1-70B: 13.5 (#304)

BenchmarkClaude 3 HaikuLlama 3.1-70B
OTIS Mock AIME 2024-20251.8%3.6%
LMArena Math11881252
MATH Level 514.9%36.7%
Omni-MATH—21%

Knowledge Llama 3.1-70B leads

Claude 3 Haiku: 17.3 (#285), Llama 3.1-70B: 24.2 (#269)

BenchmarkClaude 3 HaikuLlama 3.1-70B
GPQA Diamond36.3%44.2%
LMArena Expert11481209
MMLU73.8%80.1%
MMLU-Pro—65.3%
Confabulations34.2%—
GPQA (HELM)—42.6%

Multimodal Not comparable

Claude 3 Haiku: 23.6 (#128), Llama 3.1-70B: —

BenchmarkClaude 3 HaikuLlama 3.1-70B
LMArena Vision950—
ScienceQA72%—

Multilingual Llama 3.1-70B leads

Claude 3 Haiku: 36.0 (#243), Llama 3.1-70B: 38.8 (#225)

BenchmarkClaude 3 HaikuLlama 3.1-70B
LMArena Non-English11781219
LMArena Chinese11551215
LMArena French11951261
LMArena German11741222
LMArena Japanese11021132
LMArena Korean11091140
LMArena Russian12041234
LMArena Spanish11661253

Instruction Following Llama 3.1-70B leads

Claude 3 Haiku: 61.3 (#247), Llama 3.1-70B: 65.3 (#223)

BenchmarkClaude 3 HaikuLlama 3.1-70B
LMArena Instruction Following11731231
IFEval—82.1%

Long Context Llama 3.1-70B leads

Claude 3 Haiku: 36.1 (#237), Llama 3.1-70B: 37.6 (#214)

BenchmarkClaude 3 HaikuLlama 3.1-70B
LMArena Longer Query11901241

Writing & Preference Llama 3.1-70B leads

Claude 3 Haiku: 29.7 (#291), Llama 3.1-70B: 35.4 (#267)

BenchmarkClaude 3 HaikuLlama 3.1-70B
LMArena Text11951261
LMArena Creative Writing11571232
EQ-Bench Creative Writing717784
LMArena Multi-Turn11901256
WildBench—75.8%

Frequently asked questions

Is Claude 3 Haiku better than Llama 3.1-70B?

Llama 3.1-70B is the stronger model overall, scoring 29.6 to 25.9 on the Noometry Index.

Is Claude 3 Haiku or Llama 3.1-70B better for coding?

Llama 3.1-70B scores higher on coding benchmarks: 30.3 versus 26.4 in the Noometry coding category.

How many benchmarks do Claude 3 Haiku and Llama 3.1-70B share?

28 benchmarks have published results for both models. Claude 3 Haiku has 37 scored results on Noometry and Llama 3.1-70B has 35.

Related comparisons

Go deeper