Noometry

Model comparison

Gemini 1.0 Pro vs Llama 3.1-8B

Gemini 1.0 Pro is the stronger model overall, scoring 27.1 to 22.8 on the Noometry Index.

Last verified . 24 shared benchmarks.

Gemini 1.0 Pro Google

27.1

Rank #327 Confirmed

Llama 3.1-8B Meta

22.8

Rank #347 Confirmed

Summary

  • They share 24 benchmarks with published results for both. Gemini 1.0 Pro scores higher in 4 categories and Llama 3.1-8B in 4 categories; 6 gaps are clear of the uncertainty.
  • The widest gap is in coding, where Gemini 1.0 Pro leads 31.6 to 19.9.
  • The biggest single-benchmark swing is MATH Level 5: 11.2% for Gemini 1.0 Pro and 22.9% for Llama 3.1-8B.
  • Llama 3.1-8B has downloadable open weights; the other is API-only.

Side by side

Gemini 1.0 ProLlama 3.1-8B
ProviderGoogleMeta
Noometry Index27.122.8
Released2023-12-132024-07-23
WeightsProprietaryOpen
Context window—128K
Max output—4K
Input $ / M tokens—$0.05
Output $ / M tokens—$0.08
Results tracked2443

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemini 1.0 Pro leads

Gemini 1.0 Pro: 31.6 (#270), Llama 3.1-8B: 19.9 (#335)

BenchmarkGemini 1.0 ProLlama 3.1-8B
LMArena Coding11081195
HumanEval+55.5%62.8%
MBPP+61.4%55.6%
SciCode—13.2%
WeirdML—1.7%
BigCodeBench Instruct—32.8%
BigCodeBench Complete—40.5%

Agentic & Tool Use Not comparable

Gemini 1.0 Pro: —, Llama 3.1-8B: 22.3 (#123)

BenchmarkGemini 1.0 ProLlama 3.1-8B
Berkeley Function Calling Leaderboard—25.8%
BALROG—15.1%

Reasoning Gemini 1.0 Pro leads

Gemini 1.0 Pro: 16.4 (#292), Llama 3.1-8B: 14.5 (#316)

BenchmarkGemini 1.0 ProLlama 3.1-8B
LMArena Hard Prompts11091175
DTBench45.9%50.9%
Epoch Capabilities Index116.97116.48
CritPt—0%
Chess Puzzles—0%
LMCA—5.4%
PIQA—81.2%

Math Too close to call

Gemini 1.0 Pro: 8.7 (#315), Llama 3.1-8B: 9.5 (#311)

BenchmarkGemini 1.0 ProLlama 3.1-8B
OTIS Mock AIME 2024-20251.1%1.7%
LMArena Math11321179
MATH Level 511.2%22.9%
Omni-MATH—13.7%
GSM8K—82.4%

Knowledge Gemini 1.0 Pro leads

Gemini 1.0 Pro: 15.5 (#286), Llama 3.1-8B: 8.0 (#302)

BenchmarkGemini 1.0 ProLlama 3.1-8B
GPQA Diamond34%27%
LMArena Expert10591144
MMLU70%56.1%
MMLU-Pro—40.6%
GPQA (HELM)—24.7%
BoolQ—82.8%

Multilingual Too close to call

Gemini 1.0 Pro: 33.5 (#247), Llama 3.1-8B: 34.2 (#244)

BenchmarkGemini 1.0 ProLlama 3.1-8B
LMArena Non-English11381148
LMArena Chinese11241151
LMArena French11441176
LMArena German11251144
LMArena Japanese10231061
LMArena Russian11861158
LMArena Spanish11181169
LMArena Korean—1053

Instruction Following Llama 3.1-8B leads

Gemini 1.0 Pro: 57.9 (#262), Llama 3.1-8B: 59.1 (#253)

BenchmarkGemini 1.0 ProLlama 3.1-8B
LMArena Instruction Following11141159
IFEval—74.3%

Long Context Llama 3.1-8B leads

Gemini 1.0 Pro: 34.3 (#244), Llama 3.1-8B: 35.8 (#233)

BenchmarkGemini 1.0 ProLlama 3.1-8B
LMArena Longer Query11321182

Writing & Preference Gemini 1.0 Pro leads

Gemini 1.0 Pro: 36.2 (#259), Llama 3.1-8B: 29.9 (#285)

BenchmarkGemini 1.0 ProLlama 3.1-8B
LMArena Text11491187
LMArena Creative Writing11301154
LMArena Multi-Turn11391172
EQ-Bench Creative Writing—713
WildBench—68.7%

Frequently asked questions

Is Gemini 1.0 Pro better than Llama 3.1-8B?

Gemini 1.0 Pro is the stronger model overall, scoring 27.1 to 22.8 on the Noometry Index.

Is Gemini 1.0 Pro or Llama 3.1-8B better for coding?

Gemini 1.0 Pro scores higher on coding benchmarks: 31.6 versus 19.9 in the Noometry coding category.

How many benchmarks do Gemini 1.0 Pro and Llama 3.1-8B share?

24 benchmarks have published results for both models. Gemini 1.0 Pro has 24 scored results on Noometry and Llama 3.1-8B has 43.

Related comparisons

Go deeper