Noometry

Model comparison

GPT-4 Turbo vs StarCoder 2 7B

GPT-4 Turbo has enough public results to be ranked (#281); StarCoder 2 7B does not yet, so treat this comparison as directional.

Last verified . 4 shared benchmarks.

GPT-4 Turbo OpenAI

30.3

Rank #281 Confirmed

StarCoder 2 7B NVIDIA

36.5

Unranked Sparse

Summary

  • They share 4 benchmarks with published results for both. GPT-4 Turbo scores higher in 0 categories and StarCoder 2 7B in 1 category, but none of those gaps is larger than the uncertainty.
  • The biggest single-benchmark swing is BigCodeBench Complete: 58.2% for GPT-4 Turbo and 27.7% for StarCoder 2 7B.
  • StarCoder 2 7B has downloadable open weights; the other is API-only.

Side by side

GPT-4 TurboStarCoder 2 7B
ProviderOpenAINVIDIA
Noometry Index30.336.5
Released2023-11-062024-02-20
WeightsProprietaryOpen
Context window128K—
Max output4K—
Input $ / M tokens$10—
Output $ / M tokens$30—
Results tracked367

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Too close to call

GPT-4 Turbo: 33.4 (#247), StarCoder 2 7B: 34.2

BenchmarkGPT-4 TurboStarCoder 2 7B
BigCodeBench Complete58.2%27.7%
HumanEval+86.6%29.9%
WeirdML18%—
BigCodeBench Instruct48.2%—
LMArena Coding1268—
MBPP+73.3%—

Agentic & Tool Use Not comparable

GPT-4 Turbo: —, StarCoder 2 7B: —

BenchmarkGPT-4 TurboStarCoder 2 7B
METR Time Horizons36.7%—

Reasoning Not comparable

GPT-4 Turbo: 14.9 (#311), StarCoder 2 7B: —

BenchmarkGPT-4 TurboStarCoder 2 7B
Epoch Capabilities Index127.2592.92
SimpleBench25.1%—
Chess Puzzles6%—
LMArena Hard Prompts1251—
DTBench61.6%—
LMCA9.8%—
ForecastBench59.4—
WinoGrande—57.1%

Math Not comparable

GPT-4 Turbo: 8.6 (#316), StarCoder 2 7B: —

BenchmarkGPT-4 TurboStarCoder 2 7B
FrontierMath (Tiers 1-3)0.7%—
OTIS Mock AIME 2024-20256.7%—
LMArena Math1272—
MATH Level 546.7%—
GSM8K—32.7%

Knowledge Not comparable

GPT-4 Turbo: 24.3 (#263), StarCoder 2 7B: —

BenchmarkGPT-4 TurboStarCoder 2 7B
MMLU81.3%38.8%
GPQA Diamond46.6%—
Confabulations28.4%—
LMArena Expert1223—
ARC (AI2) Challenge—38.7%

Multimodal Not comparable

GPT-4 Turbo: 30.0 (#104), StarCoder 2 7B: —

BenchmarkGPT-4 TurboStarCoder 2 7B
LMArena Vision1090—

Multilingual Not comparable

GPT-4 Turbo: 40.7 (#211), StarCoder 2 7B: —

BenchmarkGPT-4 TurboStarCoder 2 7B
LMArena Non-English1245—
LMArena Chinese1242—
LMArena French1275—
LMArena German1259—
LMArena Japanese1194—
LMArena Korean1187—
LMArena Russian1259—
LMArena Spanish1260—

Instruction Following Not comparable

GPT-4 Turbo: 66.0 (#211), StarCoder 2 7B: —

BenchmarkGPT-4 TurboStarCoder 2 7B
LMArena Instruction Following1250—

Long Context Not comparable

GPT-4 Turbo: 38.0 (#200), StarCoder 2 7B: —

BenchmarkGPT-4 TurboStarCoder 2 7B
LMArena Longer Query1254—

Writing & Preference Not comparable

GPT-4 Turbo: 47.9 (#201), StarCoder 2 7B: —

BenchmarkGPT-4 TurboStarCoder 2 7B
LMArena Text1272—
LMArena Creative Writing1269—
LMArena Multi-Turn1267—

Frequently asked questions

Is GPT-4 Turbo better than StarCoder 2 7B?

GPT-4 Turbo has enough public results to be ranked (#281); StarCoder 2 7B does not yet, so treat this comparison as directional.

Is GPT-4 Turbo or StarCoder 2 7B better for coding?

They score almost the same on coding (33.4 vs 34.2); test both on your own repository before choosing.

How many benchmarks do GPT-4 Turbo and StarCoder 2 7B share?

4 benchmarks have published results for both models. GPT-4 Turbo has 36 scored results on Noometry and StarCoder 2 7B has 7.

Related comparisons

Go deeper