Noometry

Model comparison

Claude 3.5 Haiku vs Gemma 3 1B

Claude 3.5 Haiku is the stronger model overall, scoring 28.9 to 20.7 on the Noometry Index.

Last verified . 2 shared benchmarks.

Claude 3.5 Haiku Anthropic

28.9

Rank #309 Confirmed

Gemma 3 1B Google

20.7

Rank #348 Reported

Summary

  • They share 2 benchmarks with published results for both. Claude 3.5 Haiku scores higher in 3 categories and Gemma 3 1B in 1 category; 4 gaps are clear of the uncertainty.
  • The widest gap is in agentic & tool use, where Claude 3.5 Haiku leads 27.7 to 13.5.
  • The biggest single-benchmark swing is GPQA Diamond: 38.1% for Claude 3.5 Haiku and 19.9% for Gemma 3 1B.
  • Gemma 3 1B has downloadable open weights; the other is API-only.

Side by side

Claude 3.5 HaikuGemma 3 1B
ProviderAnthropicGoogle
Noometry Index28.920.7
Released2024-10-222025-03-12
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked494

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Claude 3.5 Haiku: 32.6 (#258), Gemma 3 1B: —

BenchmarkClaude 3.5 HaikuGemma 3 1B
Aider Polyglot28%—
SciCode27.4%—
WeirdML30.7%—
BigCodeBench Instruct46.1%—
LiveBench Coding51.4%—
LMArena Coding1287—
BigCodeBench Complete59%—
CadEval32%—

Agentic & Tool Use Claude 3.5 Haiku leads

Claude 3.5 Haiku: 27.7 (#90), Gemma 3 1B: 13.5 (#144)

BenchmarkClaude 3.5 HaikuGemma 3 1B
Berkeley Function Calling Leaderboard—7.2%
BALROG19.3%—

Reasoning Gemma 3 1B leads

Claude 3.5 Haiku: 17.1 (#286), Gemma 3 1B: 18.5 (#260)

BenchmarkClaude 3.5 HaikuGemma 3 1B
CritPt0%—
Chess Puzzles—0%
LiveBench Reasoning28.1%—
LMArena Hard Prompts1251—
DTBench56.7%—
LiveBench Data Analysis48.5%—
Epoch Capabilities Index127.15—
LiveBench43.5%—

Math Claude 3.5 Haiku leads

Claude 3.5 Haiku: 13.7 (#294), Gemma 3 1B: 9.6 (#310)

BenchmarkClaude 3.5 HaikuGemma 3 1B
OTIS Mock AIME 2024-20254.3%1.1%
Omni-MATH22.4%—
LiveBench Math35.5%—
LMArena Math1244—
MATH Level 546.4%—
FrontierMath (Feb 2025 set)0.3%—

Knowledge Claude 3.5 Haiku leads

Claude 3.5 Haiku: 18.7 (#276), Gemma 3 1B: 7.1 (#309)

BenchmarkClaude 3.5 HaikuGemma 3 1B
GPQA Diamond38.1%19.9%
MMLU-Pro60.5%—
Confabulations36.7%—
GPQA (HELM)36.3%—
LMArena Expert1208—
MMLU74.3%—

Multimodal Not comparable

Claude 3.5 Haiku: 26.3 (#113), Gemma 3 1B: —

BenchmarkClaude 3.5 HaikuGemma 3 1B
LMArena Vision1092—
GeoBench34%—

Multilingual Not comparable

Claude 3.5 Haiku: 40.2 (#213), Gemma 3 1B: —

BenchmarkClaude 3.5 HaikuGemma 3 1B
LMArena Non-English1238—
LMArena Chinese1229—
LMArena French1262—
LMArena German1237—
LMArena Japanese1175—
LMArena Korean1172—
LMArena Russian1253—
LMArena Spanish1261—

Instruction Following Not comparable

Claude 3.5 Haiku: 63.1 (#229), Gemma 3 1B: —

BenchmarkClaude 3.5 HaikuGemma 3 1B
LiveBench Instruction Following61.9%—
IFEval79.2%—
LMArena Instruction Following1241—

Long Context Not comparable

Claude 3.5 Haiku: 38.3 (#195), Gemma 3 1B: —

BenchmarkClaude 3.5 HaikuGemma 3 1B
LMArena Longer Query1261—

Writing & Preference Not comparable

Claude 3.5 Haiku: 42.9 (#228), Gemma 3 1B: —

BenchmarkClaude 3.5 HaikuGemma 3 1B
LMArena Text1255—
LMArena Creative Writing1233—
Short-Story Creative Writing73.5%—
EQ-Bench Creative Writing1146—
WildBench76%—
LMArena Multi-Turn1265—
LiveBench Language35.4%—

Frequently asked questions

Is Claude 3.5 Haiku better than Gemma 3 1B?

Claude 3.5 Haiku is the stronger model overall, scoring 28.9 to 20.7 on the Noometry Index.

How many benchmarks do Claude 3.5 Haiku and Gemma 3 1B share?

2 benchmarks have published results for both models. Claude 3.5 Haiku has 49 scored results on Noometry and Gemma 3 1B has 4.

Related comparisons

Go deeper