Noometry

Model comparison

Claude 3 Haiku vs Gemini 1.5 Flash 8B

Gemini 1.5 Flash 8B is the stronger model overall, scoring 29.7 to 25.7 on the Noometry Index.

Last verified . 21 shared benchmarks.

Claude 3 Haiku Anthropic

25.7

Rank #333 Confirmed

Gemini 1.5 Flash 8B Google

29.7

Rank #292 Confirmed

Summary

  • They share 21 benchmarks with published results for both. Claude 3 Haiku scores higher in 1 category and Gemini 1.5 Flash 8B in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Gemini 1.5 Flash 8B leads 43.0 to 29.8.

Side by side

Claude 3 HaikuGemini 1.5 Flash 8B
ProviderAnthropicGoogle
Noometry Index25.729.7
Released2024-03-072024-10-03
WeightsProprietaryProprietary
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked3721

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemini 1.5 Flash 8B leads

Claude 3 Haiku: 26.1 (#320), Gemini 1.5 Flash 8B: 34.9 (#221)

BenchmarkClaude 3 HaikuGemini 1.5 Flash 8B
LMArena Coding11991218
WeirdML9.8%—
BigCodeBench Instruct39.4%—
BigCodeBench Complete50.1%—
CadEval12%—
HumanEval+68.9%—
MBPP+68.8%—

Reasoning Gemini 1.5 Flash 8B leads

Claude 3 Haiku: 15.7 (#303), Gemini 1.5 Flash 8B: 19.2 (#242)

BenchmarkClaude 3 HaikuGemini 1.5 Flash 8B
LMArena Hard Prompts11741210
DTBench50.1%50%
Kagi LLM Benchmark34.2%—
LMCA8.8%—
Epoch Capabilities Index118.3—
ForecastBench53.2—
WinoGrande74.2%—

Math Gemini 1.5 Flash 8B leads

Claude 3 Haiku: 9.2 (#313), Gemini 1.5 Flash 8B: 13.3 (#296)

BenchmarkClaude 3 HaikuGemini 1.5 Flash 8B
OTIS Mock AIME 2024-20251.8%4.6%
LMArena Math11881207
MATH Level 514.9%—

Knowledge Claude 3 Haiku leads

Claude 3 Haiku: 17.3 (#280), Gemini 1.5 Flash 8B: 16.0 (#285)

BenchmarkClaude 3 HaikuGemini 1.5 Flash 8B
GPQA Diamond36.3%33%
LMArena Expert11481185
Confabulations34.2%—
MMLU73.8%—

Multimodal Gemini 1.5 Flash 8B leads

Claude 3 Haiku: 23.1 (#123), Gemini 1.5 Flash 8B: 27.6 (#110)

BenchmarkClaude 3 HaikuGemini 1.5 Flash 8B
LMArena Vision9501044
ScienceQA72%—

Multilingual Gemini 1.5 Flash 8B leads

Claude 3 Haiku: 36.1 (#238), Gemini 1.5 Flash 8B: 38.6 (#224)

BenchmarkClaude 3 HaikuGemini 1.5 Flash 8B
LMArena Non-English11781215
LMArena Chinese11551231
LMArena French11941233
LMArena German11741206
LMArena Japanese11021150
LMArena Korean11081140
LMArena Russian12041236
LMArena Spanish11651212

Instruction Following Gemini 1.5 Flash 8B leads

Claude 3 Haiku: 61.5 (#242), Gemini 1.5 Flash 8B: 63.1 (#231)

BenchmarkClaude 3 HaikuGemini 1.5 Flash 8B
LMArena Instruction Following11741199

Long Context Too close to call

Claude 3 Haiku: 36.1 (#232), Gemini 1.5 Flash 8B: 36.9 (#220)

BenchmarkClaude 3 HaikuGemini 1.5 Flash 8B
LMArena Longer Query11911219

Writing & Preference Gemini 1.5 Flash 8B leads

Claude 3 Haiku: 29.8 (#286), Gemini 1.5 Flash 8B: 43.0 (#227)

BenchmarkClaude 3 HaikuGemini 1.5 Flash 8B
LMArena Text11951226
LMArena Creative Writing11571218
LMArena Multi-Turn11901185
EQ-Bench Creative Writing717—

Frequently asked questions

Is Claude 3 Haiku better than Gemini 1.5 Flash 8B?

Gemini 1.5 Flash 8B is the stronger model overall, scoring 29.7 to 25.7 on the Noometry Index.

Is Claude 3 Haiku or Gemini 1.5 Flash 8B better for coding?

Gemini 1.5 Flash 8B scores higher on coding benchmarks: 34.9 versus 26.1 in the Noometry coding category.

How many benchmarks do Claude 3 Haiku and Gemini 1.5 Flash 8B share?

21 benchmarks have published results for both models. Claude 3 Haiku has 37 scored results on Noometry and Gemini 1.5 Flash 8B has 21.

Related comparisons

Go deeper