Noometry

Model comparison

Gemma 1.1 7b IT vs StarCoder 2 15B

Gemma 1.1 7b IT has enough public results to be ranked (#275); StarCoder 2 15B does not yet, so treat this comparison as directional.

Last verified . 1 shared benchmarks.

Gemma 1.1 7b IT Google

30.7

Rank #275 Confirmed

StarCoder 2 15B NVIDIA

37.5

Unranked Sparse

Summary

  • They share 1 benchmark with published results for both. Gemma 1.1 7b IT scores higher in 0 categories and StarCoder 2 15B in 1 category; one gap is clear of the uncertainty.
  • The widest gap is in coding, where StarCoder 2 15B leads 36.0 to 31.0.

Side by side

Gemma 1.1 7b ITStarCoder 2 15B
ProviderGoogleNVIDIA
Noometry Index30.737.5
Released—2024-02-20
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked197

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding StarCoder 2 15B leads

Gemma 1.1 7b IT: 31.0 (#279), StarCoder 2 15B: 36.0

BenchmarkGemma 1.1 7b ITStarCoder 2 15B
HumanEval+35.4%37.8%
LMArena Coding1084—
BigCodeBench Complete—38.4%
MBPP+45%—

Reasoning Not comparable

Gemma 1.1 7b IT: 19.6 (#235), StarCoder 2 15B: —

BenchmarkGemma 1.1 7b ITStarCoder 2 15B
LMArena Hard Prompts1071—
Epoch Capabilities Index—104.61
WinoGrande—64.3%

Math Not comparable

Gemma 1.1 7b IT: 29.5 (#217), StarCoder 2 15B: —

BenchmarkGemma 1.1 7b ITStarCoder 2 15B
LMArena Math1107—
GSM8K—57.7%

Knowledge Not comparable

Gemma 1.1 7b IT: 27.8 (#242), StarCoder 2 15B: —

BenchmarkGemma 1.1 7b ITStarCoder 2 15B
LMArena Expert1039—
ARC (AI2) Challenge—47.2%
MMLU—64.1%

Multilingual Not comparable

Gemma 1.1 7b IT: 28.3 (#268), StarCoder 2 15B: —

BenchmarkGemma 1.1 7b ITStarCoder 2 15B
LMArena Non-English1052—
LMArena Chinese1060—
LMArena French1064—
LMArena German1054—
LMArena Japanese971—
LMArena Korean988—
LMArena Russian1046—
LMArena Spanish1049—

Instruction Following Not comparable

Gemma 1.1 7b IT: 54.3 (#278), StarCoder 2 15B: —

BenchmarkGemma 1.1 7b ITStarCoder 2 15B
LMArena Instruction Following1057—

Long Context Not comparable

Gemma 1.1 7b IT: 32.0 (#267), StarCoder 2 15B: —

BenchmarkGemma 1.1 7b ITStarCoder 2 15B
LMArena Longer Query1056—

Writing & Preference Not comparable

Gemma 1.1 7b IT: 30.6 (#283), StarCoder 2 15B: —

BenchmarkGemma 1.1 7b ITStarCoder 2 15B
LMArena Text1094—
LMArena Creative Writing1059—
LMArena Multi-Turn1040—

Frequently asked questions

Is Gemma 1.1 7b IT better than StarCoder 2 15B?

Gemma 1.1 7b IT has enough public results to be ranked (#275); StarCoder 2 15B does not yet, so treat this comparison as directional.

Is Gemma 1.1 7b IT or StarCoder 2 15B better for coding?

StarCoder 2 15B scores higher on coding benchmarks: 36.0 versus 31.0 in the Noometry coding category.

How many benchmarks do Gemma 1.1 7b IT and StarCoder 2 15B share?

1 benchmark has published results for both models. Gemma 1.1 7b IT has 19 scored results on Noometry and StarCoder 2 15B has 7.

Related comparisons

Go deeper