Noometry

Model comparison

Llama 3.1-8B vs o1-mini

o1-mini is the stronger model overall, scoring 34.0 to 23.0 on the Noometry Index.

Last verified . 24 shared benchmarks.

Llama 3.1-8B Meta

23.0

Rank #352 Confirmed

o1-mini OpenAI

34.0

Rank #235 Confirmed

Summary

  • They share 24 benchmarks with published results for both. Llama 3.1-8B scores higher in 1 category and o1-mini in 8 categories; 9 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where o1-mini leads 34.9 to 8.0.
  • The biggest single-benchmark swing is MATH Level 5: 22.9% for Llama 3.1-8B and 89.2% for o1-mini.
  • Llama 3.1-8B has downloadable open weights; the other is API-only.

Side by side

Llama 3.1-8Bo1-mini
ProviderMetaOpenAI
Noometry Index23.034.0
Released2024-07-232024-09-12
WeightsOpenProprietary
Context window128K—
Max output4K—
Input $ / M tokens$0.05—
Output $ / M tokens$0.08—
Results tracked4339

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o1-mini leads

Llama 3.1-8B: 20.2 (#340), o1-mini: 35.5 (#224)

BenchmarkLlama 3.1-8Bo1-mini
WeirdML1.7%36.3%
LMArena Coding11951362
HumanEval+62.8%89%
MBPP+55.6%78.8%
Aider Polyglot—32.9%
SciCode13.2%—
BigCodeBench Instruct32.8%—
LiveBench Coding—48%
BigCodeBench Complete40.5%—

Agentic & Tool Use o1-mini leads

Llama 3.1-8B: 22.5 (#131), o1-mini: 24.6 (#118)

BenchmarkLlama 3.1-8Bo1-mini
Berkeley Function Calling Leaderboard25.8%—
Cybench—10%
BALROG15.1%—

Reasoning Llama 3.1-8B leads

Llama 3.1-8B: 14.9 (#321), o1-mini: 8.8 (#346)

BenchmarkLlama 3.1-8Bo1-mini
LMArena Hard Prompts11751333
Epoch Capabilities Index116.57135.82
ARC-AGI-2—0.8%
SimpleBench—18.1%
ARC-AGI-1—14%
CritPt0%—
Chess Puzzles0%—
LiveBench Reasoning—72.3%
DTBench50.9%—
LiveBench Data Analysis—57.9%
LMCA5.4%—
LiveBench—57.8%
PIQA81.2%—

Math o1-mini leads

Llama 3.1-8B: 10.2 (#317), o1-mini: 35.4 (#186)

BenchmarkLlama 3.1-8Bo1-mini
OTIS Mock AIME 2024-20251.7%46.9%
LMArena Math11791358
MATH Level 522.9%89.2%
Omni-MATH13.7%—
LiveBench Math—62%
FrontierMath (Feb 2025 set)—1.7%
GSM8K82.4%—

Knowledge o1-mini leads

Llama 3.1-8B: 8.0 (#307), o1-mini: 34.9 (#192)

BenchmarkLlama 3.1-8Bo1-mini
GPQA Diamond27%62.4%
LMArena Expert11441316
MMLU-Pro40.6%—
Confabulations—18.6%
GPQA (HELM)24.7%—
BoolQ82.8%—
MMLU56.1%—

Multilingual o1-mini leads

Llama 3.1-8B: 34.0 (#249), o1-mini: 43.6 (#182)

BenchmarkLlama 3.1-8Bo1-mini
LMArena Non-English11481289
LMArena Chinese11511314
LMArena French11771293
LMArena German11441278
LMArena Japanese10611245
LMArena Korean10531223
LMArena Russian11581283
LMArena Spanish11691303

Instruction Following o1-mini leads

Llama 3.1-8B: 58.9 (#258), o1-mini: 66.7 (#206)

BenchmarkLlama 3.1-8Bo1-mini
LMArena Instruction Following11591304
LiveBench Instruction Following—65.4%
IFEval74.3%—

Long Context o1-mini leads

Llama 3.1-8B: 35.8 (#238), o1-mini: 40.1 (#161)

BenchmarkLlama 3.1-8Bo1-mini
LMArena Longer Query11821320

Writing & Preference o1-mini leads

Llama 3.1-8B: 29.7 (#290), o1-mini: 48.4 (#202)

BenchmarkLlama 3.1-8Bo1-mini
LMArena Text11871317
LMArena Creative Writing11541244
LMArena Multi-Turn11721314
Short-Story Creative Writing—64.9%
EQ-Bench Creative Writing713—
WildBench68.7%—
LiveBench Language—40.9%

Frequently asked questions

Is Llama 3.1-8B better than o1-mini?

o1-mini is the stronger model overall, scoring 34.0 to 23.0 on the Noometry Index.

Is Llama 3.1-8B or o1-mini better for coding?

o1-mini scores higher on coding benchmarks: 35.5 versus 20.2 in the Noometry coding category.

How many benchmarks do Llama 3.1-8B and o1-mini share?

24 benchmarks have published results for both models. Llama 3.1-8B has 43 scored results on Noometry and o1-mini has 39.

Related comparisons

Go deeper