Noometry

Model comparison

Llama 3.1-8B vs Mercury 2.5

Mercury 2.5 is the stronger model overall, scoring 33.5 to 23.0 on the Noometry Index.

Last verified . 2 shared benchmarks.

Llama 3.1-8B Meta

23.0

Rank #352 Confirmed

Mercury 2.5 Inception

33.5

Rank #242 Reported

Summary

  • They share 2 benchmarks with published results for both. Llama 3.1-8B scores higher in 0 categories and Mercury 2.5 in 3 categories; 3 gaps are clear of the uncertainty.
  • The widest gap is in coding, where Mercury 2.5 leads 39.5 to 20.2.
  • The biggest single-benchmark swing is SciCode: 13.2% for Llama 3.1-8B and 38.5% for Mercury 2.5.
  • Llama 3.1-8B is cheaper at $0.05 / $0.08 per million input/output tokens, against $0.04 / $0.15 for Mercury 2.5.
  • Mercury 2.5 accepts more context: 260K tokens versus 128K.
  • Llama 3.1-8B has downloadable open weights; the other is API-only.

Side by side

Llama 3.1-8BMercury 2.5
ProviderMetaInception
Noometry Index23.033.5
Released2024-07-232026-09-08
WeightsOpenProprietary
Context window128K260K
Max output4K66K
Input $ / M tokens$0.05$0.04
Output $ / M tokens$0.08$0.15
Results tracked434

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mercury 2.5 leads

Llama 3.1-8B: 20.2 (#340), Mercury 2.5: 39.5 (#156)

BenchmarkLlama 3.1-8BMercury 2.5
SciCode13.2%38.5%
WeirdML1.7%—
BigCodeBench Instruct32.8%—
LMArena Coding1195—
BigCodeBench Complete40.5%—
ALE-Bench—301.65
HumanEval+62.8%—
MBPP+55.6%—

Agentic & Tool Use Not comparable

Llama 3.1-8B: 22.5 (#131), Mercury 2.5: —

BenchmarkLlama 3.1-8BMercury 2.5
Berkeley Function Calling Leaderboard25.8%—
BALROG15.1%—

Reasoning Mercury 2.5 leads

Llama 3.1-8B: 14.9 (#321), Mercury 2.5: 22.4 (#193)

BenchmarkLlama 3.1-8BMercury 2.5
CritPt0%0%
Chess Puzzles0%—
LMArena Hard Prompts1175—
DTBench50.9%—
LMCA5.4%—
Epoch Capabilities Index116.57—
PIQA81.2%—

Math Mercury 2.5 leads

Llama 3.1-8B: 10.2 (#317), Mercury 2.5: 23.3 (#272)

BenchmarkLlama 3.1-8BMercury 2.5
OTIS Mock AIME 2024-20251.7%—
ProofBench—3%
Omni-MATH13.7%—
LMArena Math1179—
MATH Level 522.9%—
GSM8K82.4%—

Knowledge Not comparable

Llama 3.1-8B: 8.0 (#307), Mercury 2.5: —

BenchmarkLlama 3.1-8BMercury 2.5
GPQA Diamond27%—
MMLU-Pro40.6%—
GPQA (HELM)24.7%—
LMArena Expert1144—
BoolQ82.8%—
MMLU56.1%—

Multilingual Not comparable

Llama 3.1-8B: 34.0 (#249), Mercury 2.5: —

BenchmarkLlama 3.1-8BMercury 2.5
LMArena Non-English1148—
LMArena Chinese1151—
LMArena French1177—
LMArena German1144—
LMArena Japanese1061—
LMArena Korean1053—
LMArena Russian1158—
LMArena Spanish1169—

Instruction Following Not comparable

Llama 3.1-8B: 58.9 (#258), Mercury 2.5: —

BenchmarkLlama 3.1-8BMercury 2.5
IFEval74.3%—
LMArena Instruction Following1159—

Long Context Not comparable

Llama 3.1-8B: 35.8 (#238), Mercury 2.5: —

BenchmarkLlama 3.1-8BMercury 2.5
LMArena Longer Query1182—

Writing & Preference Not comparable

Llama 3.1-8B: 29.7 (#290), Mercury 2.5: —

BenchmarkLlama 3.1-8BMercury 2.5
LMArena Text1187—
LMArena Creative Writing1154—
EQ-Bench Creative Writing713—
WildBench68.7%—
LMArena Multi-Turn1172—

Frequently asked questions

Is Llama 3.1-8B better than Mercury 2.5?

Mercury 2.5 is the stronger model overall, scoring 33.5 to 23.0 on the Noometry Index.

Which is cheaper, Llama 3.1-8B or Mercury 2.5?

Llama 3.1-8B is cheaper. It lists at $0.05 per million input tokens and $0.08 per million output tokens; Mercury 2.5 lists at $0.04 and $0.15.

Is Llama 3.1-8B or Mercury 2.5 better for coding?

Mercury 2.5 scores higher on coding benchmarks: 39.5 versus 20.2 in the Noometry coding category.

Which has the bigger context window?

Mercury 2.5 does, with 260K tokens against 128K.

How many benchmarks do Llama 3.1-8B and Mercury 2.5 share?

2 benchmarks have published results for both models. Llama 3.1-8B has 43 scored results on Noometry and Mercury 2.5 has 4.

Related comparisons

Go deeper