Noometry

Model comparison

Gemini 1.5 Flash 8B vs Llama 2-70B

Gemini 1.5 Flash 8B is the stronger model overall, scoring 29.9 to 24.4 on the Noometry Index.

Last verified . 20 shared benchmarks.

Gemini 1.5 Flash 8B Google

29.9

Rank #301 Confirmed

Llama 2-70B Meta

24.4

Rank #349 Confirmed

Summary

  • They share 20 benchmarks with published results for both. Gemini 1.5 Flash 8B scores higher in 8 categories and Llama 2-70B in 0 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in multilingual, where Gemini 1.5 Flash 8B leads 38.5 to 27.7.
  • The biggest single-benchmark swing is DTBench: 50% for Gemini 1.5 Flash 8B and 41.6% for Llama 2-70B.
  • Llama 2-70B has downloadable open weights; the other is API-only.

Side by side

Gemini 1.5 Flash 8BLlama 2-70B
ProviderGoogleMeta
Noometry Index29.924.4
Released2024-10-032023-07-18
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked2135

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 35.5 (#225), Llama 2-70B: 31.4 (#286)

BenchmarkGemini 1.5 Flash 8BLlama 2-70B
LMArena Coding12181079

Reasoning Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 20.0 (#244), Llama 2-70B: 14.4 (#325)

BenchmarkGemini 1.5 Flash 8BLlama 2-70B
LMArena Hard Prompts12091073
DTBench50%41.6%
BIG-Bench Hard—64.9%
CommonsenseQA 2.0—50%
Epoch Capabilities Index—113.79
ForecastBench—51.4
HellaSwag—85.3%
LAMBADA—78.9%
PIQA—82.8%
WinoGrande—80.2%

Math Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 14.2 (#302), Llama 2-70B: 8.1 (#326)

BenchmarkGemini 1.5 Flash 8BLlama 2-70B
OTIS Mock AIME 2024-20254.6%0%
LMArena Math12071091
MATH Level 5—3.3%
GSM8K—69.6%

Knowledge Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 16.0 (#289), Llama 2-70B: 7.4 (#310)

BenchmarkGemini 1.5 Flash 8BLlama 2-70B
GPQA Diamond33%26.3%
LMArena Expert11851039
ARC (AI2) Challenge—78.3%
BoolQ—88.6%
MMLU—69.9%
OpenBookQA—60.2%
TriviaQA—87.6%

Multimodal Not comparable

Gemini 1.5 Flash 8B: 28.2 (#115), Llama 2-70B: —

BenchmarkGemini 1.5 Flash 8BLlama 2-70B
LMArena Vision1044—

Multilingual Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 38.5 (#229), Llama 2-70B: 27.7 (#274)

BenchmarkGemini 1.5 Flash 8BLlama 2-70B
LMArena Non-English12151045
LMArena Chinese1231995
LMArena French12341090
LMArena German12061041
LMArena Japanese1150927
LMArena Korean1140964
LMArena Russian12361083
LMArena Spanish12121143

Instruction Following Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 62.8 (#236), Llama 2-70B: 54.9 (#278)

BenchmarkGemini 1.5 Flash 8BLlama 2-70B
LMArena Instruction Following11991071

Long Context Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 37.0 (#225), Llama 2-70B: 32.3 (#270)

BenchmarkGemini 1.5 Flash 8BLlama 2-70B
LMArena Longer Query12191062

Writing & Preference Gemini 1.5 Flash 8B leads

Gemini 1.5 Flash 8B: 42.8 (#232), Llama 2-70B: 32.3 (#279)

BenchmarkGemini 1.5 Flash 8BLlama 2-70B
LMArena Text12261115
LMArena Creative Writing12181075
LMArena Multi-Turn11851088

Frequently asked questions

Is Gemini 1.5 Flash 8B better than Llama 2-70B?

Gemini 1.5 Flash 8B is the stronger model overall, scoring 29.9 to 24.4 on the Noometry Index.

Is Gemini 1.5 Flash 8B or Llama 2-70B better for coding?

Gemini 1.5 Flash 8B scores higher on coding benchmarks: 35.5 versus 31.4 in the Noometry coding category.

How many benchmarks do Gemini 1.5 Flash 8B and Llama 2-70B share?

20 benchmarks have published results for both models. Gemini 1.5 Flash 8B has 21 scored results on Noometry and Llama 2-70B has 35.

Related comparisons

Go deeper