Noometry

Model comparison

Claude 3.5 Haiku vs Gemma 2B

Claude 3.5 Haiku and Gemma 2B score almost the same on the Noometry Index (29.2 vs 29.6), so choose on price, context window or the category you care about most.

Last verified . 13 shared benchmarks.

Claude 3.5 Haiku Anthropic

29.2

Rank #315 Confirmed

Gemma 2B Google

29.6

Rank #307 Confirmed

Summary

  • They share 13 benchmarks with published results for both. Claude 3.5 Haiku scores higher in 5 categories and Gemma 2B in 2 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Claude 3.5 Haiku leads 42.7 to 24.0.
  • Gemma 2B has downloadable open weights; the other is API-only.

Side by side

Claude 3.5 HaikuGemma 2B
ProviderAnthropicGoogle
Noometry Index29.229.6
Released2024-10-222024-02-21
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked4923

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Claude 3.5 Haiku leads

Claude 3.5 Haiku: 32.9 (#265), Gemma 2B: 29.4 (#305)

BenchmarkClaude 3.5 HaikuGemma 2B
LMArena Coding12861010
Aider Polyglot28%—
SciCode27.4%—
WeirdML30.7%—
BigCodeBench Instruct46.1%—
LiveBench Coding51.4%—
BigCodeBench Complete59%—
CadEval32%—
HumanEval+—20.7%
MBPP+—34.1%

Agentic & Tool Use Not comparable

Claude 3.5 Haiku: 28.0 (#95), Gemma 2B: —

BenchmarkClaude 3.5 HaikuGemma 2B
BALROG19.3%—

Reasoning Gemma 2B leads

Claude 3.5 Haiku: 17.7 (#290), Gemma 2B: 18.8 (#275)

BenchmarkClaude 3.5 HaikuGemma 2B
LMArena Hard Prompts1251989
Epoch Capabilities Index127.1594.2
CritPt0%—
LiveBench Reasoning28.1%—
DTBench56.7%—
LiveBench Data Analysis48.5%—
BIG-Bench Hard—35.2%
HellaSwag—71.4%
LiveBench43.5%—
PIQA—77.3%
WinoGrande—65.4%

Math Gemma 2B leads

Claude 3.5 Haiku: 14.7 (#300), Gemma 2B: 30.0 (#239)

BenchmarkClaude 3.5 HaikuGemma 2B
LMArena Math12441009
OTIS Mock AIME 2024-20254.3%—
Omni-MATH22.4%—
LiveBench Math35.5%—
MATH Level 546.4%—
FrontierMath (Feb 2025 set)0.3%—
GSM8K—17.7%

Knowledge Not comparable

Claude 3.5 Haiku: 18.7 (#281), Gemma 2B: —

BenchmarkClaude 3.5 HaikuGemma 2B
MMLU74.3%42.3%
GPQA Diamond38.1%—
MMLU-Pro60.5%—
Confabulations36.7%—
GPQA (HELM)36.3%—
LMArena Expert1208—
ARC (AI2) Challenge—42.1%
BoolQ—69.4%
TriviaQA—53.2%

Multimodal Not comparable

Claude 3.5 Haiku: 26.8 (#117), Gemma 2B: —

BenchmarkClaude 3.5 HaikuGemma 2B
LMArena Vision1092—
GeoBench34%—

Multilingual Claude 3.5 Haiku leads

Claude 3.5 Haiku: 40.0 (#218), Gemma 2B: 23.0 (#294)

BenchmarkClaude 3.5 HaikuGemma 2B
LMArena Non-English1238958
LMArena Chinese1229986
LMArena Russian1253937
LMArena French1264—
LMArena German1237—
LMArena Japanese1175—
LMArena Korean1173—
LMArena Spanish1261—

Instruction Following Claude 3.5 Haiku leads

Claude 3.5 Haiku: 62.9 (#234), Gemma 2B: 48.5 (#302)

BenchmarkClaude 3.5 HaikuGemma 2B
LMArena Instruction Following1241970
LiveBench Instruction Following61.9%—
IFEval79.2%—

Long Context Claude 3.5 Haiku leads

Claude 3.5 Haiku: 38.3 (#200), Gemma 2B: 29.9 (#291)

BenchmarkClaude 3.5 HaikuGemma 2B
LMArena Longer Query1261981

Writing & Preference Claude 3.5 Haiku leads

Claude 3.5 Haiku: 42.7 (#234), Gemma 2B: 24.0 (#308)

BenchmarkClaude 3.5 HaikuGemma 2B
LMArena Text12551002
LMArena Creative Writing1233987
LMArena Multi-Turn1265945
Short-Story Creative Writing73.5%—
EQ-Bench Creative Writing1146—
WildBench76%—
LiveBench Language35.4%—

Frequently asked questions

Is Claude 3.5 Haiku better than Gemma 2B?

Claude 3.5 Haiku and Gemma 2B score almost the same on the Noometry Index (29.2 vs 29.6), so choose on price, context window or the category you care about most.

Is Claude 3.5 Haiku or Gemma 2B better for coding?

Claude 3.5 Haiku scores higher on coding benchmarks: 32.9 versus 29.4 in the Noometry coding category.

How many benchmarks do Claude 3.5 Haiku and Gemma 2B share?

13 benchmarks have published results for both models. Claude 3.5 Haiku has 49 scored results on Noometry and Gemma 2B has 23.

Related comparisons

Go deeper