Noometry

Model comparison

Llama 3.1-70B vs Mercury 2

Mercury 2 is the stronger model overall, scoring 39.1 to 29.6 on the Noometry Index.

Last verified . 12 shared benchmarks.

Llama 3.1-70B Meta

29.6

Rank #308 Confirmed

Mercury 2 Inception

39.1

Rank #175 Confirmed

Summary

  • They share 12 benchmarks with published results for both. Llama 3.1-70B scores higher in 0 categories and Mercury 2 in 7 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Mercury 2 leads 53.8 to 35.4.
  • The biggest single-benchmark swing is WeirdML: 9% for Llama 3.1-70B and 43.2% for Mercury 2.
  • Mercury 2 is cheaper at $0.25 / $0.75 per million input/output tokens, against $0.40 / $0.40 for Llama 3.1-70B.
  • Llama 3.1-70B has downloadable open weights; the other is API-only.

Side by side

Llama 3.1-70BMercury 2
ProviderMetaInception
Noometry Index29.639.1
Released2024-07-232026-02-20
WeightsOpenProprietary
Context window128K128K
Max output4K50K
Input $ / M tokens$0.40$0.25
Output $ / M tokens$0.40$0.75
Results tracked3517

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Mercury 2 leads

Llama 3.1-70B: 30.3 (#296), Mercury 2: 33.5 (#255)

BenchmarkLlama 3.1-70BMercury 2
WeirdML9%43.2%
LMArena Coding12601391
LMArena WebDev—1171
SciCode—38.7%
BigCodeBench Instruct46.1%—
BigCodeBench Complete54.8%—
ALE-Bench—785.58

Agentic & Tool Use Not comparable

Llama 3.1-70B: 25.1 (#112), Mercury 2: —

BenchmarkLlama 3.1-70BMercury 2
TheAgentCompany6.9%—
BALROG27.9%—

Reasoning Mercury 2 leads

Llama 3.1-70B: 21.6 (#220), Mercury 2: 23.8 (#170)

BenchmarkLlama 3.1-70BMercury 2
LMArena Hard Prompts12411362
CritPt—0.8%
DTBench60%—
LMCA14.8%—
Epoch Capabilities Index125.92—

Math Not comparable

Llama 3.1-70B: 13.5 (#304), Mercury 2: —

BenchmarkLlama 3.1-70BMercury 2
OTIS Mock AIME 2024-20253.6%—
Omni-MATH21%—
LMArena Math1252—
MATH Level 536.7%—

Knowledge Mercury 2 leads

Llama 3.1-70B: 24.2 (#269), Mercury 2: 36.2 (#172)

BenchmarkLlama 3.1-70BMercury 2
LMArena Expert12091358
GPQA Diamond44.2%—
MMLU-Pro65.3%—
Vectara Hallucination Rate—12.3%
GPQA (HELM)42.6%—
MMLU80.1%—

Multilingual Mercury 2 leads

Llama 3.1-70B: 38.8 (#225), Mercury 2: 46.6 (#157)

BenchmarkLlama 3.1-70BMercury 2
LMArena Non-English12191331
LMArena Chinese12151417
LMArena Russian12341304
LMArena French1261—
LMArena German1222—
LMArena Japanese1132—
LMArena Korean1140—
LMArena Spanish1253—

Instruction Following Mercury 2 leads

Llama 3.1-70B: 65.3 (#223), Mercury 2: 70.2 (#165)

BenchmarkLlama 3.1-70BMercury 2
LMArena Instruction Following12311329
IFEval82.1%—

Long Context Mercury 2 leads

Llama 3.1-70B: 37.6 (#214), Mercury 2: 40.5 (#154)

BenchmarkLlama 3.1-70BMercury 2
LMArena Longer Query12411330

Writing & Preference Mercury 2 leads

Llama 3.1-70B: 35.4 (#267), Mercury 2: 53.8 (#155)

BenchmarkLlama 3.1-70BMercury 2
LMArena Text12611355
LMArena Creative Writing12321289
LMArena Multi-Turn12561358
EQ-Bench Creative Writing784—
WildBench75.8%—

Frequently asked questions

Is Llama 3.1-70B better than Mercury 2?

Mercury 2 is the stronger model overall, scoring 39.1 to 29.6 on the Noometry Index.

Which is cheaper, Llama 3.1-70B or Mercury 2?

Mercury 2 is cheaper. It lists at $0.25 per million input tokens and $0.75 per million output tokens; Llama 3.1-70B lists at $0.40 and $0.40.

Is Llama 3.1-70B or Mercury 2 better for coding?

Mercury 2 scores higher on coding benchmarks: 33.5 versus 30.3 in the Noometry coding category.

Which has the bigger context window?

Both accept 128K tokens.

How many benchmarks do Llama 3.1-70B and Mercury 2 share?

12 benchmarks have published results for both models. Llama 3.1-70B has 35 scored results on Noometry and Mercury 2 has 17.

Related comparisons

Go deeper