Noometry

Model comparison

Gemma 3 12B vs Phi-4

Gemma 3 12B and Phi-4 score almost the same on the Noometry Index (32.1 vs 31.2), so choose on price, context window or the category you care about most.

Last verified . 18 shared benchmarks.

Gemma 3 12B Google

32.1

Rank #262 Confirmed

Phi-4 Microsoft

31.2

Rank #279 Confirmed

Summary

  • They share 18 benchmarks with published results for both. Gemma 3 12B scores higher in 6 categories and Phi-4 in 3 categories; 9 gaps are clear of the uncertainty.
  • The widest gap is in multilingual, where Gemma 3 12B leads 45.7 to 37.2.
  • The biggest single-benchmark swing is GPQA Diamond: 39.5% for Gemma 3 12B and 56.1% for Phi-4.
  • Gemma 3 12B is cheaper at $0.05 / $0.15 per million input/output tokens, against $0.07 / $0.14 for Phi-4.
  • Gemma 3 12B accepts more context: 131K tokens versus 128K.

Side by side

Gemma 3 12BPhi-4
ProviderGoogleMicrosoft
Noometry Index32.131.2
Released2025-03-122024-12-11
WeightsOpenOpen
Context window131K128K
Max output8K4K
Input $ / M tokens$0.05$0.07
Output $ / M tokens$0.15$0.14
Results tracked2437

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Phi-4 leads

Gemma 3 12B: 31.7 (#280), Phi-4: 34.4 (#239)

BenchmarkGemma 3 12BPhi-4
LMArena Coding12811231
SciCode17.4%—
BigCodeBench Instruct—45.5%
LiveBench Coding—30.7%
BigCodeBench Complete—55.4%

Agentic & Tool Use Gemma 3 12B leads

Gemma 3 12B: 25.5 (#108), Phi-4: 22.8 (#128)

BenchmarkGemma 3 12BPhi-4
Berkeley Function Calling Leaderboard30.4%28.8%
BALROG—11.6%

Reasoning Phi-4 leads

Gemma 3 12B: 15.7 (#313), Phi-4: 17.7 (#291)

BenchmarkGemma 3 12BPhi-4
Chess Puzzles0%1%
LMArena Hard Prompts13091220
Epoch Capabilities Index123.5130.42
CritPt0%—
LiveBench Reasoning—47.8%
DTBench48.8%—
LiveBench Data Analysis—45.2%
LMCA4.5%—
LiveBench—41.6%

Math Gemma 3 12B leads

Gemma 3 12B: 22.3 (#279), Phi-4: 20.8 (#285)

BenchmarkGemma 3 12BPhi-4
OTIS Mock AIME 2024-202516.7%13.8%
LMArena Math13071246
LiveBench Math—42%
MATH Level 5—64.9%

Knowledge Phi-4 leads

Gemma 3 12B: 26.5 (#257), Phi-4: 32.6 (#209)

BenchmarkGemma 3 12BPhi-4
GPQA Diamond39.5%56.1%
Vectara Hallucination Rate4.4%3.7%
LMArena Expert12481203
Confabulations—29.4%
MMLU—84.8%

Multimodal Not comparable

Gemma 3 12B: —, Phi-4: —

BenchmarkGemma 3 12BPhi-4
MindCube46.7%—

Multilingual Gemma 3 12B leads

Gemma 3 12B: 45.7 (#165), Phi-4: 37.2 (#237)

BenchmarkGemma 3 12BPhi-4
LMArena Non-English13181197
LMArena German13701222
LMArena Russian13351209
LMArena Chinese—1212
LMArena French—1224
LMArena Japanese—1158
LMArena Korean—1151
LMArena Spanish—1234

Instruction Following Gemma 3 12B leads

Gemma 3 12B: 68.6 (#186), Phi-4: 60.4 (#251)

BenchmarkGemma 3 12BPhi-4
LMArena Instruction Following12991201
LiveBench Instruction Following—58.4%

Long Context Gemma 3 12B leads

Gemma 3 12B: 40.0 (#162), Phi-4: 36.9 (#226)

BenchmarkGemma 3 12BPhi-4
LMArena Longer Query13171217

Writing & Preference Gemma 3 12B leads

Gemma 3 12B: 47.5 (#209), Phi-4: 40.5 (#244)

BenchmarkGemma 3 12BPhi-4
LMArena Text13341217
LMArena Creative Writing13311182
LMArena Multi-Turn13341206
Short-Story Creative Writing—62.6%
EQ-Bench Creative Writing1126—
LiveBench Language—25.6%

Frequently asked questions

Is Gemma 3 12B better than Phi-4?

Gemma 3 12B and Phi-4 score almost the same on the Noometry Index (32.1 vs 31.2), so choose on price, context window or the category you care about most.

Which is cheaper, Gemma 3 12B or Phi-4?

Gemma 3 12B is cheaper. It lists at $0.05 per million input tokens and $0.15 per million output tokens; Phi-4 lists at $0.07 and $0.14.

Is Gemma 3 12B or Phi-4 better for coding?

Phi-4 scores higher on coding benchmarks: 34.4 versus 31.7 in the Noometry coding category.

Which has the bigger context window?

Gemma 3 12B does, with 131K tokens against 128K.

How many benchmarks do Gemma 3 12B and Phi-4 share?

18 benchmarks have published results for both models. Gemma 3 12B has 24 scored results on Noometry and Phi-4 has 37.

Related comparisons

Go deeper