Noometry

Model comparison

Claude 3 Haiku vs Llama 2-13B

Llama 2-13B is the stronger model overall, scoring 29.6 to 25.9 on the Noometry Index.

Last verified . 22 shared benchmarks.

Claude 3 Haiku Anthropic

25.9

Rank #340 Confirmed

Llama 2-13B Meta

29.6

Rank #309 Confirmed

Summary

  • They share 22 benchmarks with published results for both. Claude 3 Haiku scores higher in 4 categories and Llama 2-13B in 4 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in math, where Llama 2-13B leads 31.1 to 9.8.
  • The biggest single-benchmark swing is DTBench: 50.1% for Claude 3 Haiku and 42.2% for Llama 2-13B.
  • Llama 2-13B has downloadable open weights; the other is API-only.

Side by side

Claude 3 HaikuLlama 2-13B
ProviderAnthropicMeta
Noometry Index25.929.6
Released2024-03-072023-07-18
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked3732

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Llama 2-13B leads

Claude 3 Haiku: 26.4 (#325), Llama 2-13B: 30.9 (#291)

BenchmarkClaude 3 HaikuLlama 2-13B
LMArena Coding11991062
WeirdML9.8%—
BigCodeBench Instruct39.4%—
BigCodeBench Complete50.1%—
CadEval12%—
HumanEval+68.9%—
MBPP+68.8%—

Reasoning Claude 3 Haiku leads

Claude 3 Haiku: 16.3 (#307), Llama 2-13B: 12.8 (#337)

BenchmarkClaude 3 HaikuLlama 2-13B
LMArena Hard Prompts11741051
DTBench50.1%42.2%
Epoch Capabilities Index118.35106.17
WinoGrande74.2%72.8%
Kagi LLM Benchmark34.2%—
Chess Puzzles—0%
LMCA8.8%—
BIG-Bench Hard—58.2%
ForecastBench53.2—
HellaSwag—80.7%
LAMBADA—76.5%
PIQA—80.8%

Math Llama 2-13B leads

Claude 3 Haiku: 9.8 (#319), Llama 2-13B: 31.1 (#229)

BenchmarkClaude 3 HaikuLlama 2-13B
LMArena Math11881065
OTIS Mock AIME 2024-20251.8%—
MATH Level 514.9%—
GSM8K—36.9%

Knowledge Llama 2-13B leads

Claude 3 Haiku: 17.3 (#285), Llama 2-13B: 28.1 (#249)

BenchmarkClaude 3 HaikuLlama 2-13B
LMArena Expert11481030
MMLU73.8%55.6%
GPQA Diamond36.3%—
Confabulations34.2%—
ARC (AI2) Challenge—60.3%
BoolQ—82.4%
OpenBookQA—57%
TriviaQA—79.6%

Multimodal Not comparable

Claude 3 Haiku: 23.6 (#128), Llama 2-13B: —

BenchmarkClaude 3 HaikuLlama 2-13B
ScienceQA72%55.8%
LMArena Vision950—

Multilingual Claude 3 Haiku leads

Claude 3 Haiku: 36.0 (#243), Llama 2-13B: 26.5 (#279)

BenchmarkClaude 3 HaikuLlama 2-13B
LMArena Non-English11781024
LMArena Chinese11551001
LMArena French11951044
LMArena German11741009
LMArena Japanese1102894
LMArena Korean1109953
LMArena Russian12041055
LMArena Spanish11661087

Instruction Following Claude 3 Haiku leads

Claude 3 Haiku: 61.3 (#247), Llama 2-13B: 53.3 (#287)

BenchmarkClaude 3 HaikuLlama 2-13B
LMArena Instruction Following11731045

Long Context Claude 3 Haiku leads

Claude 3 Haiku: 36.1 (#237), Llama 2-13B: 32.3 (#269)

BenchmarkClaude 3 HaikuLlama 2-13B
LMArena Longer Query11901064

Writing & Preference Too close to call

Claude 3 Haiku: 29.7 (#291), Llama 2-13B: 29.8 (#289)

BenchmarkClaude 3 HaikuLlama 2-13B
LMArena Text11951084
LMArena Creative Writing11571047
LMArena Multi-Turn11901050
EQ-Bench Creative Writing717—

Frequently asked questions

Is Claude 3 Haiku better than Llama 2-13B?

Llama 2-13B is the stronger model overall, scoring 29.6 to 25.9 on the Noometry Index.

Is Claude 3 Haiku or Llama 2-13B better for coding?

Llama 2-13B scores higher on coding benchmarks: 30.9 versus 26.4 in the Noometry coding category.

How many benchmarks do Claude 3 Haiku and Llama 2-13B share?

22 benchmarks have published results for both models. Claude 3 Haiku has 37 scored results on Noometry and Llama 2-13B has 32.

Related comparisons

Go deeper