Noometry

Model comparison

Gemma 3 4B vs Llama 3.1-8B

Gemma 3 4B is the stronger model overall, scoring 28.1 to 23.0 on the Noometry Index.

Last verified . 20 shared benchmarks.

Gemma 3 4B Google

28.1

Rank #326 Confirmed

Llama 3.1-8B Meta

23.0

Rank #352 Confirmed

Summary

  • They share 20 benchmarks with published results for both. Gemma 3 4B scores higher in 7 categories and Llama 3.1-8B in 2 categories; 9 gaps are clear of the uncertainty.
  • The widest gap is in coding, where Gemma 3 4B leads 35.9 to 20.2.
  • The biggest single-benchmark swing is Berkeley Function Calling Leaderboard: 19.6% for Gemma 3 4B and 25.8% for Llama 3.1-8B.
  • Gemma 3 4B is cheaper at $0.04 / $0.08 per million input/output tokens, against $0.05 / $0.08 for Llama 3.1-8B.
  • Gemma 3 4B accepts more context: 131K tokens versus 128K.

Side by side

Gemma 3 4BLlama 3.1-8B
ProviderGoogleMeta
Noometry Index28.123.0
Released2025-03-122024-07-23
WeightsOpenOpen
Context window131K128K
Max output4K4K
Input $ / M tokens$0.04$0.05
Output $ / M tokens$0.08$0.08
Results tracked2243

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemma 3 4B leads

Gemma 3 4B: 35.9 (#215), Llama 3.1-8B: 20.2 (#340)

BenchmarkGemma 3 4BLlama 3.1-8B
LMArena Coding12301195
SciCode—13.2%
WeirdML—1.7%
BigCodeBench Instruct—32.8%
BigCodeBench Complete—40.5%
HumanEval+—62.8%
MBPP+—55.6%

Agentic & Tool Use Llama 3.1-8B leads

Gemma 3 4B: 20.9 (#142), Llama 3.1-8B: 22.5 (#131)

BenchmarkGemma 3 4BLlama 3.1-8B
Berkeley Function Calling Leaderboard19.6%25.8%
BALROG—15.1%

Reasoning Llama 3.1-8B leads

Gemma 3 4B: 13.2 (#335), Llama 3.1-8B: 14.9 (#321)

BenchmarkGemma 3 4BLlama 3.1-8B
Chess Puzzles0%0%
LMArena Hard Prompts12531175
DTBench50.9%50.9%
LMCA2.8%5.4%
Epoch Capabilities Index116.02116.57
Kagi LLM Benchmark25.2%—
CritPt—0%
PIQA—81.2%

Math Gemma 3 4B leads

Gemma 3 4B: 16.8 (#292), Llama 3.1-8B: 10.2 (#317)

BenchmarkGemma 3 4BLlama 3.1-8B
OTIS Mock AIME 2024-20257.5%1.7%
LMArena Math12391179
Omni-MATH—13.7%
MATH Level 5—22.9%
GSM8K—82.4%

Knowledge Gemma 3 4B leads

Gemma 3 4B: 11.8 (#299), Llama 3.1-8B: 8.0 (#307)

BenchmarkGemma 3 4BLlama 3.1-8B
GPQA Diamond23.2%27%
LMArena Expert12231144
MMLU-Pro—40.6%
Vectara Hallucination Rate6.4%—
GPQA (HELM)—24.7%
BoolQ—82.8%
MMLU—56.1%

Multilingual Gemma 3 4B leads

Gemma 3 4B: 42.5 (#194), Llama 3.1-8B: 34.0 (#249)

BenchmarkGemma 3 4BLlama 3.1-8B
LMArena Non-English12731148
LMArena German12811144
LMArena Russian12941158
LMArena Chinese—1151
LMArena French—1177
LMArena Japanese—1061
LMArena Korean—1053
LMArena Spanish—1169

Instruction Following Gemma 3 4B leads

Gemma 3 4B: 65.2 (#225), Llama 3.1-8B: 58.9 (#258)

BenchmarkGemma 3 4BLlama 3.1-8B
LMArena Instruction Following12391159
IFEval—74.3%

Long Context Gemma 3 4B leads

Gemma 3 4B: 38.7 (#194), Llama 3.1-8B: 35.8 (#238)

BenchmarkGemma 3 4BLlama 3.1-8B
LMArena Longer Query12731182

Writing & Preference Gemma 3 4B leads

Gemma 3 4B: 42.0 (#239), Llama 3.1-8B: 29.7 (#290)

BenchmarkGemma 3 4BLlama 3.1-8B
LMArena Text12911187
LMArena Creative Writing12711154
EQ-Bench Creative Writing1068713
LMArena Multi-Turn12551172
WildBench—68.7%

Frequently asked questions

Is Gemma 3 4B better than Llama 3.1-8B?

Gemma 3 4B is the stronger model overall, scoring 28.1 to 23.0 on the Noometry Index.

Which is cheaper, Gemma 3 4B or Llama 3.1-8B?

Gemma 3 4B is cheaper. It lists at $0.04 per million input tokens and $0.08 per million output tokens; Llama 3.1-8B lists at $0.05 and $0.08.

Is Gemma 3 4B or Llama 3.1-8B better for coding?

Gemma 3 4B scores higher on coding benchmarks: 35.9 versus 20.2 in the Noometry coding category.

Which has the bigger context window?

Gemma 3 4B does, with 131K tokens against 128K.

How many benchmarks do Gemma 3 4B and Llama 3.1-8B share?

20 benchmarks have published results for both models. Gemma 3 4B has 22 scored results on Noometry and Llama 3.1-8B has 43.

Related comparisons

Go deeper