Noometry

Model comparison

Llama 3.1-70B vs Olmo 3.1 32b Instruct

Olmo 3.1 32b Instruct is the stronger model overall, scoring 39.4 to 29.6 on the Noometry Index.

Last verified . 16 shared benchmarks.

Llama 3.1-70B Meta

29.6

Rank #308 Confirmed

Olmo 3.1 32b Instruct Allen Institute for AI (Ai2)

39.4

Rank #168 Confirmed

Summary

  • They share 16 benchmarks with published results for both. Llama 3.1-70B scores higher in 0 categories and Olmo 3.1 32b Instruct in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where Olmo 3.1 32b Instruct leads 36.3 to 13.5.

Side by side

Llama 3.1-70BOlmo 3.1 32b Instruct
ProviderMetaAllen Institute for AI (Ai2)
Noometry Index29.639.4
Released2024-07-23—
WeightsOpenOpen
Context window128K—
Max output4K—
Input $ / M tokens$0.40—
Output $ / M tokens$0.40—
Results tracked3516

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Olmo 3.1 32b Instruct leads

Llama 3.1-70B: 30.3 (#296), Olmo 3.1 32b Instruct: 39.5 (#157)

BenchmarkLlama 3.1-70BOlmo 3.1 32b Instruct
LMArena Coding12601347
WeirdML9%—
BigCodeBench Instruct46.1%—
BigCodeBench Complete54.8%—

Agentic & Tool Use Not comparable

Llama 3.1-70B: 25.1 (#112), Olmo 3.1 32b Instruct: —

BenchmarkLlama 3.1-70BOlmo 3.1 32b Instruct
TheAgentCompany6.9%—
BALROG27.9%—

Reasoning Olmo 3.1 32b Instruct leads

Llama 3.1-70B: 21.6 (#220), Olmo 3.1 32b Instruct: 26.4 (#132)

BenchmarkLlama 3.1-70BOlmo 3.1 32b Instruct
LMArena Hard Prompts12411322
DTBench60%—
LMCA14.8%—
Epoch Capabilities Index125.92—

Math Olmo 3.1 32b Instruct leads

Llama 3.1-70B: 13.5 (#304), Olmo 3.1 32b Instruct: 36.3 (#167)

BenchmarkLlama 3.1-70BOlmo 3.1 32b Instruct
LMArena Math12521305
OTIS Mock AIME 2024-20253.6%—
Omni-MATH21%—
MATH Level 536.7%—

Knowledge Olmo 3.1 32b Instruct leads

Llama 3.1-70B: 24.2 (#269), Olmo 3.1 32b Instruct: 36.1 (#175)

BenchmarkLlama 3.1-70BOlmo 3.1 32b Instruct
LMArena Expert12091308
GPQA Diamond44.2%—
MMLU-Pro65.3%—
GPQA (HELM)42.6%—
MMLU80.1%—

Multilingual Olmo 3.1 32b Instruct leads

Llama 3.1-70B: 38.8 (#225), Olmo 3.1 32b Instruct: 42.6 (#191)

BenchmarkLlama 3.1-70BOlmo 3.1 32b Instruct
LMArena Non-English12191275
LMArena Chinese12151304
LMArena French12611328
LMArena German12221282
LMArena Korean11401206
LMArena Russian12341268
LMArena Spanish12531336
LMArena Japanese1132—

Instruction Following Olmo 3.1 32b Instruct leads

Llama 3.1-70B: 65.3 (#223), Olmo 3.1 32b Instruct: 68.6 (#187)

BenchmarkLlama 3.1-70BOlmo 3.1 32b Instruct
LMArena Instruction Following12311299
IFEval82.1%—

Long Context Olmo 3.1 32b Instruct leads

Llama 3.1-70B: 37.6 (#214), Olmo 3.1 32b Instruct: 39.9 (#166)

BenchmarkLlama 3.1-70BOlmo 3.1 32b Instruct
LMArena Longer Query12411312

Writing & Preference Olmo 3.1 32b Instruct leads

Llama 3.1-70B: 35.4 (#267), Olmo 3.1 32b Instruct: 50.2 (#185)

BenchmarkLlama 3.1-70BOlmo 3.1 32b Instruct
LMArena Text12611311
LMArena Creative Writing12321264
LMArena Multi-Turn12561309
EQ-Bench Creative Writing784—
WildBench75.8%—

Frequently asked questions

Is Llama 3.1-70B better than Olmo 3.1 32b Instruct?

Olmo 3.1 32b Instruct is the stronger model overall, scoring 39.4 to 29.6 on the Noometry Index.

Is Llama 3.1-70B or Olmo 3.1 32b Instruct better for coding?

Olmo 3.1 32b Instruct scores higher on coding benchmarks: 39.5 versus 30.3 in the Noometry coding category.

How many benchmarks do Llama 3.1-70B and Olmo 3.1 32b Instruct share?

16 benchmarks have published results for both models. Llama 3.1-70B has 35 scored results on Noometry and Olmo 3.1 32b Instruct has 16.

Related comparisons

Go deeper