Noometry

Model comparison

Claude 3.5 Haiku vs Gemma 2 9B

Claude 3.5 Haiku is the stronger model overall, scoring 29.2 to 25.9 on the Noometry Index.

Last verified . 32 shared benchmarks.

Claude 3.5 Haiku Anthropic

29.2

Rank #315 Confirmed

Gemma 2 9B Google

25.9

Rank #341 Confirmed

Summary

  • They share 32 benchmarks with published results for both. Claude 3.5 Haiku scores higher in 8 categories and Gemma 2 9B in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Claude 3.5 Haiku leads 42.7 to 32.1.
  • The biggest single-benchmark swing is LiveBench Coding: 51.4% for Claude 3.5 Haiku and 22.5% for Gemma 2 9B.
  • Gemma 2 9B has downloadable open weights; the other is API-only.

Side by side

Claude 3.5 HaikuGemma 2 9B
ProviderAnthropicGoogle
Noometry Index29.225.9
Released2024-10-222024-06-24
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked4935

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Claude 3.5 Haiku leads

Claude 3.5 Haiku: 32.9 (#265), Gemma 2 9B: 29.4 (#304)

BenchmarkClaude 3.5 HaikuGemma 2 9B
BigCodeBench Instruct46.1%34.7%
LiveBench Coding51.4%22.5%
LMArena Coding12861173
BigCodeBench Complete59%40.6%
Aider Polyglot28%—
SciCode27.4%—
WeirdML30.7%—
CadEval32%—

Agentic & Tool Use Not comparable

Claude 3.5 Haiku: 28.0 (#95), Gemma 2 9B: —

BenchmarkClaude 3.5 HaikuGemma 2 9B
BALROG19.3%—

Reasoning Claude 3.5 Haiku leads

Claude 3.5 Haiku: 17.7 (#290), Gemma 2 9B: 15.9 (#309)

BenchmarkClaude 3.5 HaikuGemma 2 9B
LiveBench Reasoning28.1%15.2%
LMArena Hard Prompts12511171
LiveBench Data Analysis48.5%36.4%
Epoch Capabilities Index127.15119.83
LiveBench43.5%28.7%
CritPt0%—
DTBench56.7%—
PIQA—83.7%

Math Claude 3.5 Haiku leads

Claude 3.5 Haiku: 14.7 (#300), Gemma 2 9B: 9.9 (#318)

BenchmarkClaude 3.5 HaikuGemma 2 9B
OTIS Mock AIME 2024-20254.3%0.6%
LiveBench Math35.5%19.8%
LMArena Math12441183
MATH Level 546.4%21%
Omni-MATH22.4%—
FrontierMath (Feb 2025 set)0.3%—
GSM8K—84.9%

Knowledge Claude 3.5 Haiku leads

Claude 3.5 Haiku: 18.7 (#281), Gemma 2 9B: 9.7 (#305)

BenchmarkClaude 3.5 HaikuGemma 2 9B
GPQA Diamond38.1%27.5%
LMArena Expert12081147
MMLU74.3%72.1%
MMLU-Pro60.5%—
Confabulations36.7%—
GPQA (HELM)36.3%—
BoolQ—85.7%

Multimodal Not comparable

Claude 3.5 Haiku: 26.8 (#117), Gemma 2 9B: —

BenchmarkClaude 3.5 HaikuGemma 2 9B
LMArena Vision1092—
GeoBench34%—

Multilingual Claude 3.5 Haiku leads

Claude 3.5 Haiku: 40.0 (#218), Gemma 2 9B: 36.6 (#238)

BenchmarkClaude 3.5 HaikuGemma 2 9B
LMArena Non-English12381188
LMArena Chinese12291185
LMArena French12641190
LMArena German12371186
LMArena Japanese11751144
LMArena Korean11731137
LMArena Russian12531200
LMArena Spanish12611200

Instruction Following Claude 3.5 Haiku leads

Claude 3.5 Haiku: 62.9 (#234), Gemma 2 9B: 57.6 (#269)

BenchmarkClaude 3.5 HaikuGemma 2 9B
LiveBench Instruction Following61.9%52.6%
LMArena Instruction Following12411178
IFEval79.2%—

Long Context Claude 3.5 Haiku leads

Claude 3.5 Haiku: 38.3 (#200), Gemma 2 9B: 36.3 (#233)

BenchmarkClaude 3.5 HaikuGemma 2 9B
LMArena Longer Query12611197

Writing & Preference Claude 3.5 Haiku leads

Claude 3.5 Haiku: 42.7 (#234), Gemma 2 9B: 32.1 (#281)

BenchmarkClaude 3.5 HaikuGemma 2 9B
LMArena Text12551207
LMArena Creative Writing12331206
EQ-Bench Creative Writing1146841
LMArena Multi-Turn12651193
LiveBench Language35.4%25.5%
Short-Story Creative Writing73.5%—
WildBench76%—

Frequently asked questions

Is Claude 3.5 Haiku better than Gemma 2 9B?

Claude 3.5 Haiku is the stronger model overall, scoring 29.2 to 25.9 on the Noometry Index.

Is Claude 3.5 Haiku or Gemma 2 9B better for coding?

Claude 3.5 Haiku scores higher on coding benchmarks: 32.9 versus 29.4 in the Noometry coding category.

How many benchmarks do Claude 3.5 Haiku and Gemma 2 9B share?

32 benchmarks have published results for both models. Claude 3.5 Haiku has 49 scored results on Noometry and Gemma 2 9B has 35.

Related comparisons

Go deeper