Noometry

Model comparison

Claude 2.1 vs Gemma 1.1 7b IT

Gemma 1.1 7b IT is the stronger model overall, scoring 31.3 to 25.2 on the Noometry Index.

Last verified . 0 shared benchmarks.

Claude 2.1 Anthropic

25.2

Rank #345 Reported

Gemma 1.1 7b IT Google

31.3

Rank #277 Confirmed

Summary

  • The widest gap is in math, where Gemma 1.1 7b IT leads 32.0 to 10.2.
  • Gemma 1.1 7b IT has downloadable open weights; the other is API-only.

Side by side

Claude 2.1Gemma 1.1 7b IT
ProviderAnthropicGoogle
Noometry Index25.231.3
Released2023-11-21—
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked719

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemma 1.1 7b IT leads

Claude 2.1: 26.2 (#327), Gemma 1.1 7b IT: 31.5 (#284)

BenchmarkClaude 2.1Gemma 1.1 7b IT
WeirdML7.1%—
LMArena Coding—1084
HumanEval+—35.4%
MBPP+—45%

Reasoning Too close to call

Claude 2.1: 21.4 (#221), Gemma 1.1 7b IT: 20.5 (#238)

BenchmarkClaude 2.1Gemma 1.1 7b IT
LMArena Hard Prompts—1071
DTBench51%—
Epoch Capabilities Index119.27—
ForecastBench54.2—

Math Gemma 1.1 7b IT leads

Claude 2.1: 10.2 (#315), Gemma 1.1 7b IT: 32.0 (#220)

BenchmarkClaude 2.1Gemma 1.1 7b IT
OTIS Mock AIME 2024-20251.9%—
LMArena Math—1107

Knowledge Gemma 1.1 7b IT leads

Claude 2.1: 15.4 (#292), Gemma 1.1 7b IT: 28.3 (#247)

BenchmarkClaude 2.1Gemma 1.1 7b IT
GPQA Diamond33%—
LMArena Expert—1039
MMLU73.5%—

Multilingual Not comparable

Claude 2.1: —, Gemma 1.1 7b IT: 28.1 (#273)

BenchmarkClaude 2.1Gemma 1.1 7b IT
LMArena Non-English—1052
LMArena Chinese—1061
LMArena French—1065
LMArena German—1054
LMArena Japanese—971
LMArena Korean—988
LMArena Russian—1046
LMArena Spanish—1049

Instruction Following Not comparable

Claude 2.1: —, Gemma 1.1 7b IT: 54.0 (#283)

BenchmarkClaude 2.1Gemma 1.1 7b IT
LMArena Instruction Following—1057

Long Context Not comparable

Claude 2.1: —, Gemma 1.1 7b IT: 32.1 (#272)

BenchmarkClaude 2.1Gemma 1.1 7b IT
LMArena Longer Query—1056

Writing & Preference Not comparable

Claude 2.1: —, Gemma 1.1 7b IT: 30.4 (#288)

BenchmarkClaude 2.1Gemma 1.1 7b IT
LMArena Text—1094
LMArena Creative Writing—1060
LMArena Multi-Turn—1040

Frequently asked questions

Is Claude 2.1 better than Gemma 1.1 7b IT?

Gemma 1.1 7b IT is the stronger model overall, scoring 31.3 to 25.2 on the Noometry Index.

Is Claude 2.1 or Gemma 1.1 7b IT better for coding?

Gemma 1.1 7b IT scores higher on coding benchmarks: 31.5 versus 26.2 in the Noometry coding category.

How many benchmarks do Claude 2.1 and Gemma 1.1 7b IT share?

0 benchmarks have published results for both models. Claude 2.1 has 7 scored results on Noometry and Gemma 1.1 7b IT has 19.

Related comparisons

Go deeper