Noometry

Model comparison

GPT-3.5-turbo vs Llama 3-8B

Llama 3-8B is the stronger model overall, scoring 25.5 to 23.2 on the Noometry Index.

Last verified . 34 shared benchmarks.

GPT-3.5-turbo OpenAI

23.2

Rank #350 Confirmed

Llama 3-8B Meta

25.5

Rank #344 Confirmed

Summary

  • They share 34 benchmarks with published results for both. GPT-3.5-turbo scores higher in 2 categories and Llama 3-8B in 6 categories; 4 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Llama 3-8B leads 37.5 to 25.3.
  • The biggest single-benchmark swing is BigCodeBench Complete: 50.6% for GPT-3.5-turbo and 36.9% for Llama 3-8B.
  • Llama 3-8B has downloadable open weights; the other is API-only.

Side by side

GPT-3.5-turboLlama 3-8B
ProviderOpenAIMeta
Noometry Index23.225.5
Released2023-03-012024-04-18
WeightsProprietaryOpen
Context window16K—
Max output4K—
Input $ / M tokens$0.50—
Output $ / M tokens$1.50—
Results tracked4434

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Llama 3-8B leads

GPT-3.5-turbo: 23.9 (#331), Llama 3-8B: 31.0 (#289)

BenchmarkGPT-3.5-turboLlama 3-8B
BigCodeBench Instruct39.1%31.9%
LMArena Coding11361152
BigCodeBench Complete50.6%36.9%
HumanEval+70.7%56.7%
MBPP+69.7%54.8%
WeirdML3.5%—

Agentic & Tool Use Not comparable

GPT-3.5-turbo: —, Llama 3-8B: —

BenchmarkGPT-3.5-turboLlama 3-8B
METR Time Horizons21.5%—

Reasoning Too close to call

GPT-3.5-turbo: 13.8 (#332), Llama 3-8B: 14.3 (#326)

BenchmarkGPT-3.5-turboLlama 3-8B
Chess Puzzles0%0%
LMArena Hard Prompts11081133
DTBench48.5%43.9%
Adversarial NLI58.1%57.3%
Epoch Capabilities Index118.55116.45
ForecastBench50.458.6
WinoGrande81.6%75.7%
Mystery Game Puzzles3%—
LMCA9.7%—
BIG-Bench Hard61.6%—
CommonsenseQA 2.057%—

Math Llama 3-8B leads

GPT-3.5-turbo: 6.3 (#327), Llama 3-8B: 8.8 (#323)

BenchmarkGPT-3.5-turboLlama 3-8B
OTIS Mock AIME 2024-20252.2%1.9%
LMArena Math11421151
MATH Level 515.9%6.1%
FrontierMath (Tiers 1-3)0%—
GSM8K57.8%—

Knowledge GPT-3.5-turbo leads

GPT-3.5-turbo: 10.0 (#303), Llama 3-8B: 7.8 (#308)

BenchmarkGPT-3.5-turboLlama 3-8B
GPQA Diamond28%26.1%
LMArena Expert10701113
ARC (AI2) Challenge87.4%82.8%
MMLU71.4%68.8%
OpenBookQA86%82.6%
TriviaQA85.8%67.7%
BoolQ87%—

Multilingual Too close to call

GPT-3.5-turbo: 31.5 (#258), Llama 3-8B: 30.8 (#261)

BenchmarkGPT-3.5-turboLlama 3-8B
LMArena Non-English11081098
LMArena Chinese10751076
LMArena French11181159
LMArena German10901104
LMArena Japanese1043967
LMArena Korean10191004
LMArena Russian11231109
LMArena Spanish11211173

Instruction Following Too close to call

GPT-3.5-turbo: 57.9 (#262), Llama 3-8B: 58.4 (#260)

BenchmarkGPT-3.5-turboLlama 3-8B
LMArena Instruction Following11191127

Long Context Too close to call

GPT-3.5-turbo: 34.0 (#254), Llama 3-8B: 34.2 (#251)

BenchmarkGPT-3.5-turboLlama 3-8B
LMArena Longer Query11211128

Writing & Preference Llama 3-8B leads

GPT-3.5-turbo: 25.3 (#305), Llama 3-8B: 37.5 (#256)

BenchmarkGPT-3.5-turboLlama 3-8B
LMArena Text11251166
LMArena Creative Writing10921150
LMArena Multi-Turn11171152
EQ-Bench Creative Writing451—

Frequently asked questions

Is GPT-3.5-turbo better than Llama 3-8B?

Llama 3-8B is the stronger model overall, scoring 25.5 to 23.2 on the Noometry Index.

Is GPT-3.5-turbo or Llama 3-8B better for coding?

Llama 3-8B scores higher on coding benchmarks: 31.0 versus 23.9 in the Noometry coding category.

How many benchmarks do GPT-3.5-turbo and Llama 3-8B share?

34 benchmarks have published results for both models. GPT-3.5-turbo has 44 scored results on Noometry and Llama 3-8B has 34.

Related comparisons

Go deeper