Noometry

Model comparison

Llama 3.1-8B vs Olmo 3.1 32b Think

Olmo 3.1 32b Think is the stronger model overall, scoring 37.9 to 23.0 on the Noometry Index.

Last verified . 15 shared benchmarks.

Llama 3.1-8B Meta

23.0

Rank #352 Confirmed

Olmo 3.1 32b Think Allen Institute for AI (Ai2)

37.9

Rank #191 Confirmed

Summary

  • They share 15 benchmarks with published results for both. Llama 3.1-8B scores higher in 0 categories and Olmo 3.1 32b Think in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Olmo 3.1 32b Think leads 35.7 to 8.0.

Side by side

Llama 3.1-8BOlmo 3.1 32b Think
ProviderMetaAllen Institute for AI (Ai2)
Noometry Index23.037.9
Released2024-07-23—
WeightsOpenOpen
Context window128K—
Max output4K—
Input $ / M tokens$0.05—
Output $ / M tokens$0.08—
Results tracked4315

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Olmo 3.1 32b Think leads

Llama 3.1-8B: 20.2 (#340), Olmo 3.1 32b Think: 37.7 (#189)

BenchmarkLlama 3.1-8BOlmo 3.1 32b Think
LMArena Coding11951291
SciCode13.2%—
WeirdML1.7%—
BigCodeBench Instruct32.8%—
BigCodeBench Complete40.5%—
HumanEval+62.8%—
MBPP+55.6%—

Agentic & Tool Use Not comparable

Llama 3.1-8B: 22.5 (#131), Olmo 3.1 32b Think: —

BenchmarkLlama 3.1-8BOlmo 3.1 32b Think
Berkeley Function Calling Leaderboard25.8%—
BALROG15.1%—

Reasoning Olmo 3.1 32b Think leads

Llama 3.1-8B: 14.9 (#321), Olmo 3.1 32b Think: 25.2 (#150)

BenchmarkLlama 3.1-8BOlmo 3.1 32b Think
LMArena Hard Prompts11751272
CritPt0%—
Chess Puzzles0%—
DTBench50.9%—
LMCA5.4%—
Epoch Capabilities Index116.57—
PIQA81.2%—

Math Olmo 3.1 32b Think leads

Llama 3.1-8B: 10.2 (#317), Olmo 3.1 32b Think: 36.3 (#168)

BenchmarkLlama 3.1-8BOlmo 3.1 32b Think
LMArena Math11791305
OTIS Mock AIME 2024-20251.7%—
Omni-MATH13.7%—
MATH Level 522.9%—
GSM8K82.4%—

Knowledge Olmo 3.1 32b Think leads

Llama 3.1-8B: 8.0 (#307), Olmo 3.1 32b Think: 35.7 (#181)

BenchmarkLlama 3.1-8BOlmo 3.1 32b Think
LMArena Expert11441295
GPQA Diamond27%—
MMLU-Pro40.6%—
GPQA (HELM)24.7%—
BoolQ82.8%—
MMLU56.1%—

Multilingual Olmo 3.1 32b Think leads

Llama 3.1-8B: 34.0 (#249), Olmo 3.1 32b Think: 38.1 (#231)

BenchmarkLlama 3.1-8BOlmo 3.1 32b Think
LMArena Non-English11481209
LMArena Chinese11511242
LMArena French11771260
LMArena German11441262
LMArena Russian11581193
LMArena Spanish11691289
LMArena Japanese1061—
LMArena Korean1053—

Instruction Following Olmo 3.1 32b Think leads

Llama 3.1-8B: 58.9 (#258), Olmo 3.1 32b Think: 65.6 (#218)

BenchmarkLlama 3.1-8BOlmo 3.1 32b Think
LMArena Instruction Following11591247
IFEval74.3%—

Long Context Olmo 3.1 32b Think leads

Llama 3.1-8B: 35.8 (#238), Olmo 3.1 32b Think: 38.6 (#195)

BenchmarkLlama 3.1-8BOlmo 3.1 32b Think
LMArena Longer Query11821272

Writing & Preference Olmo 3.1 32b Think leads

Llama 3.1-8B: 29.7 (#290), Olmo 3.1 32b Think: 46.2 (#220)

BenchmarkLlama 3.1-8BOlmo 3.1 32b Think
LMArena Text11871272
LMArena Creative Writing11541226
LMArena Multi-Turn11721252
EQ-Bench Creative Writing713—
WildBench68.7%—

Frequently asked questions

Is Llama 3.1-8B better than Olmo 3.1 32b Think?

Olmo 3.1 32b Think is the stronger model overall, scoring 37.9 to 23.0 on the Noometry Index.

Is Llama 3.1-8B or Olmo 3.1 32b Think better for coding?

Olmo 3.1 32b Think scores higher on coding benchmarks: 37.7 versus 20.2 in the Noometry coding category.

How many benchmarks do Llama 3.1-8B and Olmo 3.1 32b Think share?

15 benchmarks have published results for both models. Llama 3.1-8B has 43 scored results on Noometry and Olmo 3.1 32b Think has 15.

Related comparisons

Go deeper