Noometry

Model comparison

Llama 3.1-70B vs o1-mini

o1-mini is the stronger model overall, scoring 34.0 to 29.6 on the Noometry Index.

Last verified . 22 shared benchmarks.

Llama 3.1-70B Meta

29.6

Rank #308 Confirmed

o1-mini OpenAI

34.0

Rank #235 Confirmed

Summary

  • They share 22 benchmarks with published results for both. Llama 3.1-70B scores higher in 2 categories and o1-mini in 7 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where o1-mini leads 35.4 to 13.5.
  • The biggest single-benchmark swing is MATH Level 5: 36.7% for Llama 3.1-70B and 89.2% for o1-mini.
  • Llama 3.1-70B has downloadable open weights; the other is API-only.

Side by side

Llama 3.1-70Bo1-mini
ProviderMetaOpenAI
Noometry Index29.634.0
Released2024-07-232024-09-12
WeightsOpenProprietary
Context window128K—
Max output4K—
Input $ / M tokens$0.40—
Output $ / M tokens$0.40—
Results tracked3539

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o1-mini leads

Llama 3.1-70B: 30.3 (#296), o1-mini: 35.5 (#224)

BenchmarkLlama 3.1-70Bo1-mini
WeirdML9%36.3%
LMArena Coding12601362
Aider Polyglot—32.9%
BigCodeBench Instruct46.1%—
LiveBench Coding—48%
BigCodeBench Complete54.8%—
HumanEval+—89%
MBPP+—78.8%

Agentic & Tool Use Too close to call

Llama 3.1-70B: 25.1 (#112), o1-mini: 24.6 (#118)

BenchmarkLlama 3.1-70Bo1-mini
TheAgentCompany6.9%—
Cybench—10%
BALROG27.9%—

Reasoning Llama 3.1-70B leads

Llama 3.1-70B: 21.6 (#220), o1-mini: 8.8 (#346)

BenchmarkLlama 3.1-70Bo1-mini
LMArena Hard Prompts12411333
Epoch Capabilities Index125.92135.82
ARC-AGI-2—0.8%
SimpleBench—18.1%
ARC-AGI-1—14%
LiveBench Reasoning—72.3%
DTBench60%—
LiveBench Data Analysis—57.9%
LMCA14.8%—
LiveBench—57.8%

Math o1-mini leads

Llama 3.1-70B: 13.5 (#304), o1-mini: 35.4 (#186)

BenchmarkLlama 3.1-70Bo1-mini
OTIS Mock AIME 2024-20253.6%46.9%
LMArena Math12521358
MATH Level 536.7%89.2%
Omni-MATH21%—
LiveBench Math—62%
FrontierMath (Feb 2025 set)—1.7%

Knowledge o1-mini leads

Llama 3.1-70B: 24.2 (#269), o1-mini: 34.9 (#192)

BenchmarkLlama 3.1-70Bo1-mini
GPQA Diamond44.2%62.4%
LMArena Expert12091316
MMLU-Pro65.3%—
Confabulations—18.6%
GPQA (HELM)42.6%—
MMLU80.1%—

Multilingual o1-mini leads

Llama 3.1-70B: 38.8 (#225), o1-mini: 43.6 (#182)

BenchmarkLlama 3.1-70Bo1-mini
LMArena Non-English12191289
LMArena Chinese12151314
LMArena French12611293
LMArena German12221278
LMArena Japanese11321245
LMArena Korean11401223
LMArena Russian12341283
LMArena Spanish12531303

Instruction Following o1-mini leads

Llama 3.1-70B: 65.3 (#223), o1-mini: 66.7 (#206)

BenchmarkLlama 3.1-70Bo1-mini
LMArena Instruction Following12311304
LiveBench Instruction Following—65.4%
IFEval82.1%—

Long Context o1-mini leads

Llama 3.1-70B: 37.6 (#214), o1-mini: 40.1 (#161)

BenchmarkLlama 3.1-70Bo1-mini
LMArena Longer Query12411320

Writing & Preference o1-mini leads

Llama 3.1-70B: 35.4 (#267), o1-mini: 48.4 (#202)

BenchmarkLlama 3.1-70Bo1-mini
LMArena Text12611317
LMArena Creative Writing12321244
LMArena Multi-Turn12561314
Short-Story Creative Writing—64.9%
EQ-Bench Creative Writing784—
WildBench75.8%—
LiveBench Language—40.9%

Frequently asked questions

Is Llama 3.1-70B better than o1-mini?

o1-mini is the stronger model overall, scoring 34.0 to 29.6 on the Noometry Index.

Is Llama 3.1-70B or o1-mini better for coding?

o1-mini scores higher on coding benchmarks: 35.5 versus 30.3 in the Noometry coding category.

How many benchmarks do Llama 3.1-70B and o1-mini share?

22 benchmarks have published results for both models. Llama 3.1-70B has 35 scored results on Noometry and o1-mini has 39.

Related comparisons

Go deeper