Noometry

Model comparison

Gemma 3 4B vs Yi-1.5-34B

Yi-1.5-34B is the stronger model overall, scoring 30.2 to 27.9 on the Noometry Index.

Last verified . 13 shared benchmarks.

Gemma 3 4B Google

27.9

Rank #320 Confirmed

Yi-1.5-34B 01.AI

30.2

Rank #286 Confirmed

Summary

  • They share 13 benchmarks with published results for both. Gemma 3 4B scores higher in 5 categories and Yi-1.5-34B in 3 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in multilingual, where Gemma 3 4B leads 42.7 to 32.5.
  • The biggest single-benchmark swing is GPQA Diamond: 23.2% for Gemma 3 4B and 32% for Yi-1.5-34B.

Side by side

Gemma 3 4BYi-1.5-34B
ProviderGoogle01.AI
Noometry Index27.930.2
Released2025-03-122024-05-13
WeightsOpenOpen
Context window131K—
Max output4K—
Input $ / M tokens$0.04—
Output $ / M tokens$0.08—
Results tracked2221

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemma 3 4B leads

Gemma 3 4B: 35.3 (#214), Yi-1.5-34B: 31.9 (#267)

BenchmarkGemma 3 4BYi-1.5-34B
LMArena Coding12301169
BigCodeBench Instruct—33.9%
BigCodeBench Complete—43.8%

Agentic & Tool Use Not comparable

Gemma 3 4B: 20.7 (#134), Yi-1.5-34B: —

BenchmarkGemma 3 4BYi-1.5-34B
Berkeley Function Calling Leaderboard19.6%—

Reasoning Yi-1.5-34B leads

Gemma 3 4B: 12.7 (#330), Yi-1.5-34B: 21.5 (#195)

BenchmarkGemma 3 4BYi-1.5-34B
LMArena Hard Prompts12531160
Kagi LLM Benchmark25.2%—
Chess Puzzles0%—
DTBench50.9%—
LMCA2.8%—
Epoch Capabilities Index116.02—

Math Yi-1.5-34B leads

Gemma 3 4B: 15.8 (#286), Yi-1.5-34B: 25.3 (#248)

BenchmarkGemma 3 4BYi-1.5-34B
LMArena Math12391182
OTIS Mock AIME 2024-20257.5%—
MATH Level 5—25.5%

Knowledge Yi-1.5-34B leads

Gemma 3 4B: 11.8 (#294), Yi-1.5-34B: 14.8 (#290)

BenchmarkGemma 3 4BYi-1.5-34B
GPQA Diamond23.2%32%
LMArena Expert12231144
Vectara Hallucination Rate6.4%—

Multilingual Gemma 3 4B leads

Gemma 3 4B: 42.7 (#189), Yi-1.5-34B: 32.5 (#251)

BenchmarkGemma 3 4BYi-1.5-34B
LMArena Non-English12731121
LMArena German12811111
LMArena Russian12941091
LMArena Chinese—1213
LMArena French—1155
LMArena Japanese—1021
LMArena Korean—1005
LMArena Spanish—1120

Instruction Following Gemma 3 4B leads

Gemma 3 4B: 65.4 (#219), Yi-1.5-34B: 59.4 (#252)

BenchmarkGemma 3 4BYi-1.5-34B
LMArena Instruction Following12391139

Long Context Gemma 3 4B leads

Gemma 3 4B: 38.6 (#189), Yi-1.5-34B: 34.6 (#243)

BenchmarkGemma 3 4BYi-1.5-34B
LMArena Longer Query12741144

Writing & Preference Gemma 3 4B leads

Gemma 3 4B: 42.2 (#234), Yi-1.5-34B: 37.6 (#252)

BenchmarkGemma 3 4BYi-1.5-34B
LMArena Text12911173
LMArena Creative Writing12701135
LMArena Multi-Turn12551153
EQ-Bench Creative Writing1068—

Frequently asked questions

Is Gemma 3 4B better than Yi-1.5-34B?

Yi-1.5-34B is the stronger model overall, scoring 30.2 to 27.9 on the Noometry Index.

Is Gemma 3 4B or Yi-1.5-34B better for coding?

Gemma 3 4B scores higher on coding benchmarks: 35.3 versus 31.9 in the Noometry coding category.

How many benchmarks do Gemma 3 4B and Yi-1.5-34B share?

13 benchmarks have published results for both models. Gemma 3 4B has 22 scored results on Noometry and Yi-1.5-34B has 21.

Related comparisons

Go deeper