Noometry

Model comparison

GPT-3.5-turbo vs Llama 3.1-405B

Llama 3.1-405B is the stronger model overall, scoring 30.7 to 23.2 on the Noometry Index.

Last verified . 30 shared benchmarks.

GPT-3.5-turbo OpenAI

23.2

Rank #350 Confirmed

Llama 3.1-405B Meta

30.7

Rank #288 Confirmed

Summary

  • They share 30 benchmarks with published results for both. GPT-3.5-turbo scores higher in 0 categories and Llama 3.1-405B in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Llama 3.1-405B leads 30.4 to 10.0.
  • The biggest single-benchmark swing is MATH Level 5: 15.9% for GPT-3.5-turbo and 49.8% for Llama 3.1-405B.
  • Llama 3.1-405B has downloadable open weights; the other is API-only.

Side by side

GPT-3.5-turboLlama 3.1-405B
ProviderOpenAIMeta
Noometry Index23.230.7
Released2023-03-012024-07-23
WeightsProprietaryOpen
Context window16K—
Max output4K—
Input $ / M tokens$0.50—
Output $ / M tokens$1.50—
Results tracked4442

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Llama 3.1-405B leads

GPT-3.5-turbo: 23.9 (#331), Llama 3.1-405B: 33.1 (#262)

BenchmarkGPT-3.5-turboLlama 3.1-405B
WeirdML3.5%21.4%
LMArena Coding11361291
BigCodeBench Instruct39.1%—
BigCodeBench Complete50.6%—
HumanEval+70.7%—
MBPP+69.7%—

Agentic & Tool Use Not comparable

GPT-3.5-turbo: —, Llama 3.1-405B: 21.0 (#140)

BenchmarkGPT-3.5-turboLlama 3.1-405B
TheAgentCompany—7.4%
Cybench—7.5%
METR Time Horizons21.5%—

Reasoning Llama 3.1-405B leads

GPT-3.5-turbo: 13.8 (#332), Llama 3.1-405B: 16.8 (#300)

BenchmarkGPT-3.5-turboLlama 3.1-405B
LMArena Hard Prompts11081269
DTBench48.5%61.4%
BIG-Bench Hard61.6%82.9%
Epoch Capabilities Index118.55128.75
ForecastBench50.459.9
WinoGrande81.6%89.2%
SimpleBench—23%
Kagi LLM Benchmark—45%
Chess Puzzles0%—
Mystery Game Puzzles3%—
LMCA9.7%—
Adversarial NLI58.1%—
CommonsenseQA 2.057%—
HellaSwag—89.2%
PIQA—85.9%

Math Llama 3.1-405B leads

GPT-3.5-turbo: 6.3 (#327), Llama 3.1-405B: 18.4 (#290)

BenchmarkGPT-3.5-turboLlama 3.1-405B
OTIS Mock AIME 2024-20252.2%9.7%
LMArena Math11421281
MATH Level 515.9%49.8%
FrontierMath (Tiers 1-3)0%—
Omni-MATH—24.9%
GSM8K57.8%—

Knowledge Llama 3.1-405B leads

GPT-3.5-turbo: 10.0 (#303), Llama 3.1-405B: 30.4 (#227)

BenchmarkGPT-3.5-turboLlama 3.1-405B
GPQA Diamond28%50.9%
LMArena Expert10701243
ARC (AI2) Challenge87.4%95.3%
MMLU71.4%84.5%
TriviaQA85.8%82.7%
MMLU-Pro—72.3%
Confabulations—17.6%
GPQA (HELM)—52.2%
BoolQ87%—
OpenBookQA86%—

Multilingual Llama 3.1-405B leads

GPT-3.5-turbo: 31.5 (#258), Llama 3.1-405B: 40.7 (#214)

BenchmarkGPT-3.5-turboLlama 3.1-405B
LMArena Non-English11081248
LMArena Chinese10751242
LMArena French11181279
LMArena German10901252
LMArena Japanese10431208
LMArena Korean10191184
LMArena Russian11231265
LMArena Spanish11211260

Instruction Following Llama 3.1-405B leads

GPT-3.5-turbo: 57.9 (#262), Llama 3.1-405B: 65.9 (#214)

BenchmarkGPT-3.5-turboLlama 3.1-405B
LMArena Instruction Following11191259
IFEval—81.1%

Long Context Llama 3.1-405B leads

GPT-3.5-turbo: 34.0 (#254), Llama 3.1-405B: 38.4 (#197)

BenchmarkGPT-3.5-turboLlama 3.1-405B
LMArena Longer Query11211266

Writing & Preference Llama 3.1-405B leads

GPT-3.5-turbo: 25.3 (#305), Llama 3.1-405B: 38.9 (#251)

BenchmarkGPT-3.5-turboLlama 3.1-405B
LMArena Text11251284
LMArena Creative Writing10921262
EQ-Bench Creative Writing451870
LMArena Multi-Turn11171297
WildBench—78.3%

Frequently asked questions

Is GPT-3.5-turbo better than Llama 3.1-405B?

Llama 3.1-405B is the stronger model overall, scoring 30.7 to 23.2 on the Noometry Index.

Is GPT-3.5-turbo or Llama 3.1-405B better for coding?

Llama 3.1-405B scores higher on coding benchmarks: 33.1 versus 23.9 in the Noometry coding category.

How many benchmarks do GPT-3.5-turbo and Llama 3.1-405B share?

30 benchmarks have published results for both models. GPT-3.5-turbo has 44 scored results on Noometry and Llama 3.1-405B has 42.

Related comparisons

Go deeper