Noometry

Model comparison

Llama 3.1-70B vs Mistral Nemo

Llama 3.1-70B is the stronger model overall, scoring 29.6 to 26.4 on the Noometry Index. Mistral Nemo costs 2.7× less per token, which makes it the better buy when Llama 3.1-70B's lead doesn't matter for your workload.

Last verified . 6 shared benchmarks.

Llama 3.1-70B Meta

29.6

Rank #308 Confirmed

Mistral Nemo Mistral AI

26.4

Rank #337 Confirmed

Summary

  • They share 6 benchmarks with published results for both. Llama 3.1-70B scores higher in 4 categories and Mistral Nemo in 1 category; 4 gaps are clear of the uncertainty.
  • The widest gap is in math, where Mistral Nemo leads 25.5 to 13.5.
  • The biggest single-benchmark swing is MATH Level 5: 36.7% for Llama 3.1-70B and 10.8% for Mistral Nemo.
  • Mistral Nemo is cheaper at $0.15 / $0.15 per million input/output tokens, against $0.40 / $0.40 for Llama 3.1-70B.

Side by side

Llama 3.1-70BMistral Nemo
ProviderMetaMistral AI
Noometry Index29.626.4
Released2024-07-232024-07-01
WeightsOpenOpen
Context window128K128K
Max output4K128K
Input $ / M tokens$0.40$0.15
Output $ / M tokens$0.40$0.15
Results tracked3510

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Llama 3.1-70B: 30.3 (#296), Mistral Nemo: —

BenchmarkLlama 3.1-70BMistral Nemo
WeirdML9%—
BigCodeBench Instruct46.1%—
LMArena Coding1260—
BigCodeBench Complete54.8%—

Agentic & Tool Use Llama 3.1-70B leads

Llama 3.1-70B: 25.1 (#112), Mistral Nemo: 23.5 (#125)

BenchmarkLlama 3.1-70BMistral Nemo
BALROG27.9%17.6%
Berkeley Function Calling Leaderboard—27.6%
TheAgentCompany6.9%—

Reasoning Too close to call

Llama 3.1-70B: 21.6 (#220), Mistral Nemo: 20.7 (#232)

BenchmarkLlama 3.1-70BMistral Nemo
DTBench60%48.6%
Epoch Capabilities Index125.92118.68
LMArena Hard Prompts1241—
LMCA14.8%—
PIQA—83.5%

Math Mistral Nemo leads

Llama 3.1-70B: 13.5 (#304), Mistral Nemo: 25.5 (#268)

BenchmarkLlama 3.1-70BMistral Nemo
MATH Level 536.7%10.8%
OTIS Mock AIME 2024-20253.6%—
Omni-MATH21%—
LMArena Math1252—
GSM8K—84.2%

Knowledge Llama 3.1-70B leads

Llama 3.1-70B: 24.2 (#269), Mistral Nemo: 12.3 (#298)

BenchmarkLlama 3.1-70BMistral Nemo
GPQA Diamond44.2%29.9%
MMLU-Pro65.3%—
GPQA (HELM)42.6%—
LMArena Expert1209—
BoolQ—82.5%
MMLU80.1%—

Multilingual Not comparable

Llama 3.1-70B: 38.8 (#225), Mistral Nemo: —

BenchmarkLlama 3.1-70BMistral Nemo
LMArena Non-English1219—
LMArena Chinese1215—
LMArena French1261—
LMArena German1222—
LMArena Japanese1132—
LMArena Korean1140—
LMArena Russian1234—
LMArena Spanish1253—

Instruction Following Not comparable

Llama 3.1-70B: 65.3 (#223), Mistral Nemo: —

BenchmarkLlama 3.1-70BMistral Nemo
IFEval82.1%—
LMArena Instruction Following1231—

Long Context Not comparable

Llama 3.1-70B: 37.6 (#214), Mistral Nemo: —

BenchmarkLlama 3.1-70BMistral Nemo
LMArena Longer Query1241—

Writing & Preference Llama 3.1-70B leads

Llama 3.1-70B: 35.4 (#267), Mistral Nemo: 28.5 (#296)

BenchmarkLlama 3.1-70BMistral Nemo
EQ-Bench Creative Writing784881
LMArena Text1261—
LMArena Creative Writing1232—
WildBench75.8%—
LMArena Multi-Turn1256—

Frequently asked questions

Is Llama 3.1-70B better than Mistral Nemo?

Llama 3.1-70B is the stronger model overall, scoring 29.6 to 26.4 on the Noometry Index. Mistral Nemo costs 2.7× less per token, which makes it the better buy when Llama 3.1-70B's lead doesn't matter for your workload.

Which is cheaper, Llama 3.1-70B or Mistral Nemo?

Mistral Nemo is cheaper. It lists at $0.15 per million input tokens and $0.15 per million output tokens; Llama 3.1-70B lists at $0.40 and $0.40.

Which has the bigger context window?

Both accept 128K tokens.

How many benchmarks do Llama 3.1-70B and Mistral Nemo share?

6 benchmarks have published results for both models. Llama 3.1-70B has 35 scored results on Noometry and Mistral Nemo has 10.

Related comparisons

Go deeper