Noometry

Model comparison

Claude 3 Haiku vs Gemma 3 12B

Gemma 3 12B is the stronger model overall, scoring 31.8 to 25.7 on the Noometry Index.

Last verified . 18 shared benchmarks.

Claude 3 Haiku Anthropic

25.7

Rank #333 Confirmed

Gemma 3 12B Google

31.8

Rank #255 Confirmed

Summary

  • They share 18 benchmarks with published results for both. Claude 3 Haiku scores higher in 1 category and Gemma 3 12B in 7 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Gemma 3 12B leads 47.7 to 29.8.
  • The biggest single-benchmark swing is OTIS Mock AIME 2024-2025: 1.8% for Claude 3 Haiku and 16.7% for Gemma 3 12B.
  • Gemma 3 12B has downloadable open weights; the other is API-only.

Side by side

Claude 3 HaikuGemma 3 12B
ProviderAnthropicGoogle
Noometry Index25.731.8
Released2024-03-072025-03-12
WeightsProprietaryOpen
Context window—131K
Max output—8K
Input $ / M tokens—$0.05
Output $ / M tokens—$0.15
Results tracked3724

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemma 3 12B leads

Claude 3 Haiku: 26.1 (#320), Gemma 3 12B: 31.2 (#276)

BenchmarkClaude 3 HaikuGemma 3 12B
LMArena Coding11991281
SciCode—17.4%
WeirdML9.8%—
BigCodeBench Instruct39.4%—
BigCodeBench Complete50.1%—
CadEval12%—
HumanEval+68.9%—
MBPP+68.8%—

Agentic & Tool Use Not comparable

Claude 3 Haiku: —, Gemma 3 12B: 25.3 (#101)

BenchmarkClaude 3 HaikuGemma 3 12B
Berkeley Function Calling Leaderboard—30.4%

Reasoning Too close to call

Claude 3 Haiku: 15.7 (#303), Gemma 3 12B: 15.2 (#307)

BenchmarkClaude 3 HaikuGemma 3 12B
LMArena Hard Prompts11741309
DTBench50.1%48.8%
LMCA8.8%4.5%
Epoch Capabilities Index118.3123.49
Kagi LLM Benchmark34.2%—
CritPt—0%
Chess Puzzles—0%
ForecastBench53.2—
WinoGrande74.2%—

Math Gemma 3 12B leads

Claude 3 Haiku: 9.2 (#313), Gemma 3 12B: 21.0 (#273)

BenchmarkClaude 3 HaikuGemma 3 12B
OTIS Mock AIME 2024-20251.8%16.7%
LMArena Math11881307
MATH Level 514.9%—

Knowledge Gemma 3 12B leads

Claude 3 Haiku: 17.3 (#280), Gemma 3 12B: 26.5 (#250)

BenchmarkClaude 3 HaikuGemma 3 12B
GPQA Diamond36.3%39.5%
LMArena Expert11481248
Confabulations34.2%—
Vectara Hallucination Rate—4.4%
MMLU73.8%—

Multimodal Not comparable

Claude 3 Haiku: 23.1 (#123), Gemma 3 12B: —

BenchmarkClaude 3 HaikuGemma 3 12B
LMArena Vision950—
MindCube—46.7%
ScienceQA72%—

Multilingual Gemma 3 12B leads

Claude 3 Haiku: 36.1 (#238), Gemma 3 12B: 45.9 (#160)

BenchmarkClaude 3 HaikuGemma 3 12B
LMArena Non-English11781318
LMArena German11741370
LMArena Russian12041335
LMArena Chinese1155—
LMArena French1194—
LMArena Japanese1102—
LMArena Korean1108—
LMArena Spanish1165—

Instruction Following Gemma 3 12B leads

Claude 3 Haiku: 61.5 (#242), Gemma 3 12B: 68.8 (#179)

BenchmarkClaude 3 HaikuGemma 3 12B
LMArena Instruction Following11741299

Long Context Gemma 3 12B leads

Claude 3 Haiku: 36.1 (#232), Gemma 3 12B: 40.0 (#157)

BenchmarkClaude 3 HaikuGemma 3 12B
LMArena Longer Query11911317

Writing & Preference Gemma 3 12B leads

Claude 3 Haiku: 29.8 (#286), Gemma 3 12B: 47.7 (#205)

BenchmarkClaude 3 HaikuGemma 3 12B
LMArena Text11951334
LMArena Creative Writing11571331
EQ-Bench Creative Writing7171126
LMArena Multi-Turn11901334

Frequently asked questions

Is Claude 3 Haiku better than Gemma 3 12B?

Gemma 3 12B is the stronger model overall, scoring 31.8 to 25.7 on the Noometry Index.

Is Claude 3 Haiku or Gemma 3 12B better for coding?

Gemma 3 12B scores higher on coding benchmarks: 31.2 versus 26.1 in the Noometry coding category.

How many benchmarks do Claude 3 Haiku and Gemma 3 12B share?

18 benchmarks have published results for both models. Claude 3 Haiku has 37 scored results on Noometry and Gemma 3 12B has 24.

Related comparisons

Go deeper