Noometry

Model comparison

Llama 3.1-8B vs Olmo 3 32b Think

Olmo 3 32b Think is the stronger model overall, scoring 38.1 to 22.8 on the Noometry Index.

Last verified . 14 shared benchmarks.

Llama 3.1-8B Meta

22.8

Rank #347 Confirmed

Olmo 3 32b Think Allen Institute for AI (Ai2)

38.1

Rank #182 Confirmed

Summary

  • They share 14 benchmarks with published results for both. Llama 3.1-8B scores higher in 0 categories and Olmo 3 32b Think in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Olmo 3 32b Think leads 34.4 to 8.0.

Side by side

Llama 3.1-8BOlmo 3 32b Think
ProviderMetaAllen Institute for AI (Ai2)
Noometry Index22.838.1
Released2024-07-23—
WeightsOpenOpen
Context window128K—
Max output4K—
Input $ / M tokens$0.05—
Output $ / M tokens$0.08—
Results tracked4314

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Olmo 3 32b Think leads

Llama 3.1-8B: 19.9 (#335), Olmo 3 32b Think: 38.0 (#169)

BenchmarkLlama 3.1-8BOlmo 3 32b Think
LMArena Coding11951318
SciCode13.2%—
WeirdML1.7%—
BigCodeBench Instruct32.8%—
BigCodeBench Complete40.5%—
HumanEval+62.8%—
MBPP+55.6%—

Agentic & Tool Use Not comparable

Llama 3.1-8B: 22.3 (#123), Olmo 3 32b Think: —

BenchmarkLlama 3.1-8BOlmo 3 32b Think
Berkeley Function Calling Leaderboard25.8%—
BALROG15.1%—

Reasoning Olmo 3 32b Think leads

Llama 3.1-8B: 14.5 (#316), Olmo 3 32b Think: 24.9 (#138)

BenchmarkLlama 3.1-8BOlmo 3 32b Think
LMArena Hard Prompts11751302
CritPt0%—
Chess Puzzles0%—
DTBench50.9%—
LMCA5.4%—
Epoch Capabilities Index116.48—
PIQA81.2%—

Math Olmo 3 32b Think leads

Llama 3.1-8B: 9.5 (#311), Olmo 3 32b Think: 33.8 (#165)

BenchmarkLlama 3.1-8BOlmo 3 32b Think
LMArena Math11791315
OTIS Mock AIME 2024-20251.7%—
Omni-MATH13.7%—
MATH Level 522.9%—
GSM8K82.4%—

Knowledge Olmo 3 32b Think leads

Llama 3.1-8B: 8.0 (#302), Olmo 3 32b Think: 34.4 (#189)

BenchmarkLlama 3.1-8BOlmo 3 32b Think
LMArena Expert11441272
GPQA Diamond27%—
MMLU-Pro40.6%—
GPQA (HELM)24.7%—
BoolQ82.8%—
MMLU56.1%—

Multilingual Olmo 3 32b Think leads

Llama 3.1-8B: 34.2 (#244), Olmo 3 32b Think: 41.4 (#205)

BenchmarkLlama 3.1-8BOlmo 3 32b Think
LMArena Non-English11481255
LMArena Chinese11511300
LMArena French11761291
LMArena German11441290
LMArena Russian11581255
LMArena Japanese1061—
LMArena Korean1053—
LMArena Spanish1169—

Instruction Following Olmo 3 32b Think leads

Llama 3.1-8B: 59.1 (#253), Olmo 3 32b Think: 67.4 (#193)

BenchmarkLlama 3.1-8BOlmo 3 32b Think
LMArena Instruction Following11591275
IFEval74.3%—

Long Context Olmo 3 32b Think leads

Llama 3.1-8B: 35.8 (#233), Olmo 3 32b Think: 39.4 (#177)

BenchmarkLlama 3.1-8BOlmo 3 32b Think
LMArena Longer Query11821296

Writing & Preference Olmo 3 32b Think leads

Llama 3.1-8B: 29.9 (#285), Olmo 3 32b Think: 49.2 (#188)

BenchmarkLlama 3.1-8BOlmo 3 32b Think
LMArena Text11871300
LMArena Creative Writing11541256
LMArena Multi-Turn11721289
EQ-Bench Creative Writing713—
WildBench68.7%—

Frequently asked questions

Is Llama 3.1-8B better than Olmo 3 32b Think?

Olmo 3 32b Think is the stronger model overall, scoring 38.1 to 22.8 on the Noometry Index.

Is Llama 3.1-8B or Olmo 3 32b Think better for coding?

Olmo 3 32b Think scores higher on coding benchmarks: 38.0 versus 19.9 in the Noometry coding category.

How many benchmarks do Llama 3.1-8B and Olmo 3 32b Think share?

14 benchmarks have published results for both models. Llama 3.1-8B has 43 scored results on Noometry and Olmo 3 32b Think has 14.

Related comparisons

Go deeper