Noometry

Model comparison

Llama 3.1-70B vs StarCoder 2 15B

Llama 3.1-70B has enough public results to be ranked (#308); StarCoder 2 15B does not yet, so treat this comparison as directional.

Last verified . 3 shared benchmarks.

Llama 3.1-70B Meta

29.6

Rank #308 Confirmed

StarCoder 2 15B NVIDIA

38.1

Unranked Sparse

Summary

  • They share 3 benchmarks with published results for both. Llama 3.1-70B scores higher in 0 categories and StarCoder 2 15B in 1 category; one gap is clear of the uncertainty.
  • The widest gap is in coding, where StarCoder 2 15B leads 36.6 to 30.3.
  • The biggest single-benchmark swing is BigCodeBench Complete: 54.8% for Llama 3.1-70B and 38.4% for StarCoder 2 15B.

Side by side

Llama 3.1-70BStarCoder 2 15B
ProviderMetaNVIDIA
Noometry Index29.638.1
Released2024-07-232024-02-20
WeightsOpenOpen
Context window128K—
Max output4K—
Input $ / M tokens$0.40—
Output $ / M tokens$0.40—
Results tracked357

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding StarCoder 2 15B leads

Llama 3.1-70B: 30.3 (#296), StarCoder 2 15B: 36.6

BenchmarkLlama 3.1-70BStarCoder 2 15B
BigCodeBench Complete54.8%38.4%
WeirdML9%—
BigCodeBench Instruct46.1%—
LMArena Coding1260—
HumanEval+—37.8%

Agentic & Tool Use Not comparable

Llama 3.1-70B: 25.1 (#112), StarCoder 2 15B: —

BenchmarkLlama 3.1-70BStarCoder 2 15B
TheAgentCompany6.9%—
BALROG27.9%—

Reasoning Not comparable

Llama 3.1-70B: 21.6 (#220), StarCoder 2 15B: —

BenchmarkLlama 3.1-70BStarCoder 2 15B
Epoch Capabilities Index125.92105
LMArena Hard Prompts1241—
DTBench60%—
LMCA14.8%—
WinoGrande—64.3%

Math Not comparable

Llama 3.1-70B: 13.5 (#304), StarCoder 2 15B: —

BenchmarkLlama 3.1-70BStarCoder 2 15B
OTIS Mock AIME 2024-20253.6%—
Omni-MATH21%—
LMArena Math1252—
MATH Level 536.7%—
GSM8K—57.7%

Knowledge Not comparable

Llama 3.1-70B: 24.2 (#269), StarCoder 2 15B: —

BenchmarkLlama 3.1-70BStarCoder 2 15B
MMLU80.1%64.1%
GPQA Diamond44.2%—
MMLU-Pro65.3%—
GPQA (HELM)42.6%—
LMArena Expert1209—
ARC (AI2) Challenge—47.2%

Multilingual Not comparable

Llama 3.1-70B: 38.8 (#225), StarCoder 2 15B: —

BenchmarkLlama 3.1-70BStarCoder 2 15B
LMArena Non-English1219—
LMArena Chinese1215—
LMArena French1261—
LMArena German1222—
LMArena Japanese1132—
LMArena Korean1140—
LMArena Russian1234—
LMArena Spanish1253—

Instruction Following Not comparable

Llama 3.1-70B: 65.3 (#223), StarCoder 2 15B: —

BenchmarkLlama 3.1-70BStarCoder 2 15B
IFEval82.1%—
LMArena Instruction Following1231—

Long Context Not comparable

Llama 3.1-70B: 37.6 (#214), StarCoder 2 15B: —

BenchmarkLlama 3.1-70BStarCoder 2 15B
LMArena Longer Query1241—

Writing & Preference Not comparable

Llama 3.1-70B: 35.4 (#267), StarCoder 2 15B: —

BenchmarkLlama 3.1-70BStarCoder 2 15B
LMArena Text1261—
LMArena Creative Writing1232—
EQ-Bench Creative Writing784—
WildBench75.8%—
LMArena Multi-Turn1256—

Frequently asked questions

Is Llama 3.1-70B better than StarCoder 2 15B?

Llama 3.1-70B has enough public results to be ranked (#308); StarCoder 2 15B does not yet, so treat this comparison as directional.

Is Llama 3.1-70B or StarCoder 2 15B better for coding?

StarCoder 2 15B scores higher on coding benchmarks: 36.6 versus 30.3 in the Noometry coding category.

How many benchmarks do Llama 3.1-70B and StarCoder 2 15B share?

3 benchmarks have published results for both models. Llama 3.1-70B has 35 scored results on Noometry and StarCoder 2 15B has 7.

Related comparisons

Go deeper