Noometry

Model comparison

Llama 3.1-405B vs Olmo 3 32b Think

Olmo 3 32b Think is the stronger model overall, scoring 38.7 to 30.7 on the Noometry Index.

Last verified . 14 shared benchmarks.

Llama 3.1-405B Meta

30.7

Rank #288 Confirmed

Olmo 3 32b Think Allen Institute for AI (Ai2)

38.7

Rank #183 Confirmed

Summary

  • They share 14 benchmarks with published results for both. Llama 3.1-405B scores higher in 0 categories and Olmo 3 32b Think in 8 categories; 6 gaps are clear of the uncertainty.
  • The widest gap is in math, where Olmo 3 32b Think leads 36.5 to 18.4.

Side by side

Llama 3.1-405BOlmo 3 32b Think
ProviderMetaAllen Institute for AI (Ai2)
Noometry Index30.738.7
Released2024-07-23—
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked4214

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Olmo 3 32b Think leads

Llama 3.1-405B: 33.1 (#262), Olmo 3 32b Think: 38.6 (#172)

BenchmarkLlama 3.1-405BOlmo 3 32b Think
LMArena Coding12911319
WeirdML21.4%—

Agentic & Tool Use Not comparable

Llama 3.1-405B: 21.0 (#140), Olmo 3 32b Think: —

BenchmarkLlama 3.1-405BOlmo 3 32b Think
TheAgentCompany7.4%—
Cybench7.5%—

Reasoning Olmo 3 32b Think leads

Llama 3.1-405B: 16.8 (#300), Olmo 3 32b Think: 25.9 (#140)

BenchmarkLlama 3.1-405BOlmo 3 32b Think
LMArena Hard Prompts12691302
SimpleBench23%—
Kagi LLM Benchmark45%—
DTBench61.4%—
BIG-Bench Hard82.9%—
Epoch Capabilities Index128.75—
ForecastBench59.9—
HellaSwag89.2%—
PIQA85.9%—
WinoGrande89.2%—

Math Olmo 3 32b Think leads

Llama 3.1-405B: 18.4 (#290), Olmo 3 32b Think: 36.5 (#165)

BenchmarkLlama 3.1-405BOlmo 3 32b Think
LMArena Math12811316
OTIS Mock AIME 2024-20259.7%—
Omni-MATH24.9%—
MATH Level 549.8%—

Knowledge Olmo 3 32b Think leads

Llama 3.1-405B: 30.4 (#227), Olmo 3 32b Think: 35.0 (#190)

BenchmarkLlama 3.1-405BOlmo 3 32b Think
LMArena Expert12431273
GPQA Diamond50.9%—
MMLU-Pro72.3%—
Confabulations17.6%—
GPQA (HELM)52.2%—
ARC (AI2) Challenge95.3%—
MMLU84.5%—
TriviaQA82.7%—

Multilingual Too close to call

Llama 3.1-405B: 40.7 (#214), Olmo 3 32b Think: 41.2 (#210)

BenchmarkLlama 3.1-405BOlmo 3 32b Think
LMArena Non-English12481255
LMArena Chinese12421300
LMArena French12791291
LMArena German12521290
LMArena Russian12651254
LMArena Japanese1208—
LMArena Korean1184—
LMArena Spanish1260—

Instruction Following Olmo 3 32b Think leads

Llama 3.1-405B: 65.9 (#214), Olmo 3 32b Think: 67.2 (#198)

BenchmarkLlama 3.1-405BOlmo 3 32b Think
LMArena Instruction Following12591275
IFEval81.1%—

Long Context Too close to call

Llama 3.1-405B: 38.4 (#197), Olmo 3 32b Think: 39.4 (#182)

BenchmarkLlama 3.1-405BOlmo 3 32b Think
LMArena Longer Query12661296

Writing & Preference Olmo 3 32b Think leads

Llama 3.1-405B: 38.9 (#251), Olmo 3 32b Think: 49.1 (#193)

BenchmarkLlama 3.1-405BOlmo 3 32b Think
LMArena Text12841300
LMArena Creative Writing12621256
LMArena Multi-Turn12971290
EQ-Bench Creative Writing870—
WildBench78.3%—

Frequently asked questions

Is Llama 3.1-405B better than Olmo 3 32b Think?

Olmo 3 32b Think is the stronger model overall, scoring 38.7 to 30.7 on the Noometry Index.

Is Llama 3.1-405B or Olmo 3 32b Think better for coding?

Olmo 3 32b Think scores higher on coding benchmarks: 38.6 versus 33.1 in the Noometry coding category.

How many benchmarks do Llama 3.1-405B and Olmo 3 32b Think share?

14 benchmarks have published results for both models. Llama 3.1-405B has 42 scored results on Noometry and Olmo 3 32b Think has 14.

Related comparisons

Go deeper