Noometry

Model comparison

Gemini 1.0 Pro vs Llama 3.1-70B

Llama 3.1-70B is the stronger model overall, scoring 29.6 to 27.3 on the Noometry Index.

Last verified . 22 shared benchmarks.

Gemini 1.0 Pro Google

27.3

Rank #332 Confirmed

Llama 3.1-70B Meta

29.6

Rank #308 Confirmed

Summary

  • They share 22 benchmarks with published results for both. Gemini 1.0 Pro scores higher in 2 categories and Llama 3.1-70B in 6 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Llama 3.1-70B leads 24.2 to 15.6.
  • The biggest single-benchmark swing is MATH Level 5: 11.2% for Gemini 1.0 Pro and 36.7% for Llama 3.1-70B.
  • Llama 3.1-70B has downloadable open weights; the other is API-only.

Side by side

Gemini 1.0 ProLlama 3.1-70B
ProviderGoogleMeta
Noometry Index27.329.6
Released2023-12-132024-07-23
WeightsProprietaryOpen
Context window—128K
Max output—4K
Input $ / M tokens—$0.40
Output $ / M tokens—$0.40
Results tracked2435

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemini 1.0 Pro leads

Gemini 1.0 Pro: 32.2 (#275), Llama 3.1-70B: 30.3 (#296)

BenchmarkGemini 1.0 ProLlama 3.1-70B
LMArena Coding11081260
WeirdML—9%
BigCodeBench Instruct—46.1%
BigCodeBench Complete—54.8%
HumanEval+55.5%—
MBPP+61.4%—

Agentic & Tool Use Not comparable

Gemini 1.0 Pro: —, Llama 3.1-70B: 25.1 (#112)

BenchmarkGemini 1.0 ProLlama 3.1-70B
TheAgentCompany—6.9%
BALROG—27.9%

Reasoning Llama 3.1-70B leads

Gemini 1.0 Pro: 17.1 (#296), Llama 3.1-70B: 21.6 (#220)

BenchmarkGemini 1.0 ProLlama 3.1-70B
LMArena Hard Prompts11091241
DTBench45.9%60%
Epoch Capabilities Index117.04125.92
LMCA—14.8%

Math Llama 3.1-70B leads

Gemini 1.0 Pro: 9.3 (#321), Llama 3.1-70B: 13.5 (#304)

BenchmarkGemini 1.0 ProLlama 3.1-70B
OTIS Mock AIME 2024-20251.1%3.6%
LMArena Math11321252
MATH Level 511.2%36.7%
Omni-MATH—21%

Knowledge Llama 3.1-70B leads

Gemini 1.0 Pro: 15.6 (#291), Llama 3.1-70B: 24.2 (#269)

BenchmarkGemini 1.0 ProLlama 3.1-70B
GPQA Diamond34%44.2%
LMArena Expert10591209
MMLU70%80.1%
MMLU-Pro—65.3%
GPQA (HELM)—42.6%

Multilingual Llama 3.1-70B leads

Gemini 1.0 Pro: 33.4 (#252), Llama 3.1-70B: 38.8 (#225)

BenchmarkGemini 1.0 ProLlama 3.1-70B
LMArena Non-English11381219
LMArena Chinese11241215
LMArena French11451261
LMArena German11251222
LMArena Japanese10231132
LMArena Russian11861234
LMArena Spanish11191253
LMArena Korean—1140

Instruction Following Llama 3.1-70B leads

Gemini 1.0 Pro: 57.6 (#267), Llama 3.1-70B: 65.3 (#223)

BenchmarkGemini 1.0 ProLlama 3.1-70B
LMArena Instruction Following11141231
IFEval—82.1%

Long Context Llama 3.1-70B leads

Gemini 1.0 Pro: 34.3 (#249), Llama 3.1-70B: 37.6 (#214)

BenchmarkGemini 1.0 ProLlama 3.1-70B
LMArena Longer Query11321241

Writing & Preference Too close to call

Gemini 1.0 Pro: 36.0 (#264), Llama 3.1-70B: 35.4 (#267)

BenchmarkGemini 1.0 ProLlama 3.1-70B
LMArena Text11491261
LMArena Creative Writing11311232
LMArena Multi-Turn11391256
EQ-Bench Creative Writing—784
WildBench—75.8%

Frequently asked questions

Is Gemini 1.0 Pro better than Llama 3.1-70B?

Llama 3.1-70B is the stronger model overall, scoring 29.6 to 27.3 on the Noometry Index.

Is Gemini 1.0 Pro or Llama 3.1-70B better for coding?

Gemini 1.0 Pro scores higher on coding benchmarks: 32.2 versus 30.3 in the Noometry coding category.

How many benchmarks do Gemini 1.0 Pro and Llama 3.1-70B share?

22 benchmarks have published results for both models. Gemini 1.0 Pro has 24 scored results on Noometry and Llama 3.1-70B has 35.

Related comparisons

Go deeper