Noometry

Model comparison

Gemma 2 27B vs Phi-4

Phi-4 is the stronger model overall, scoring 31.2 to 29.4 on the Noometry Index.

Last verified . 32 shared benchmarks.

Gemma 2 27B Google

29.4

Rank #312 Confirmed

Phi-4 Microsoft

31.2

Rank #279 Confirmed

Summary

  • They share 32 benchmarks with published results for both. Gemma 2 27B scores higher in 4 categories and Phi-4 in 4 categories; 5 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Phi-4 leads 32.6 to 19.0.
  • The biggest single-benchmark swing is MATH Level 5: 27.9% for Gemma 2 27B and 64.9% for Phi-4.
  • Phi-4 is cheaper at $0.07 / $0.14 per million input/output tokens, against $0.65 / $0.65 for Gemma 2 27B.
  • Phi-4 accepts more context: 128K tokens versus 8K.

Side by side

Gemma 2 27BPhi-4
ProviderGoogleMicrosoft
Noometry Index29.431.2
Released2024-06-242024-12-11
WeightsOpenOpen
Context window8K128K
Max output2K4K
Input $ / M tokens$0.65$0.07
Output $ / M tokens$0.65$0.14
Results tracked3437

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Too close to call

Gemma 2 27B: 34.1 (#246), Phi-4: 34.4 (#239)

BenchmarkGemma 2 27BPhi-4
BigCodeBench Instruct42.8%45.5%
LiveBench Coding36%30.7%
LMArena Coding12111231
BigCodeBench Complete52.5%55.4%

Agentic & Tool Use Not comparable

Gemma 2 27B: —, Phi-4: 22.8 (#128)

BenchmarkGemma 2 27BPhi-4
Berkeley Function Calling Leaderboard—28.8%
BALROG—11.6%

Reasoning Phi-4 leads

Gemma 2 27B: 15.3 (#315), Phi-4: 17.7 (#291)

BenchmarkGemma 2 27BPhi-4
LiveBench Reasoning28.1%47.8%
LMArena Hard Prompts11981220
LiveBench Data Analysis47.9%45.2%
Epoch Capabilities Index122.08130.42
LiveBench38.2%41.6%
Chess Puzzles—1%
DTBench48%—
LMCA7.1%—

Math Phi-4 leads

Gemma 2 27B: 10.7 (#311), Phi-4: 20.8 (#285)

BenchmarkGemma 2 27BPhi-4
OTIS Mock AIME 2024-20251.4%13.8%
LiveBench Math26.5%42%
LMArena Math12121246
MATH Level 527.9%64.9%

Knowledge Phi-4 leads

Gemma 2 27B: 19.0 (#280), Phi-4: 32.6 (#209)

BenchmarkGemma 2 27BPhi-4
GPQA Diamond36.5%56.1%
Confabulations27.1%29.4%
LMArena Expert11721203
MMLU75.7%84.8%
Vectara Hallucination Rate—3.7%

Multilingual Gemma 2 27B leads

Gemma 2 27B: 38.6 (#226), Phi-4: 37.2 (#237)

BenchmarkGemma 2 27BPhi-4
LMArena Non-English12171197
LMArena Chinese12211212
LMArena French12471224
LMArena German12091222
LMArena Japanese11751158
LMArena Korean11741151
LMArena Russian12341209
LMArena Spanish12281234

Instruction Following Too close to call

Gemma 2 27B: 60.5 (#249), Phi-4: 60.4 (#251)

BenchmarkGemma 2 27BPhi-4
LiveBench Instruction Following58.1%58.4%
LMArena Instruction Following12061201

Long Context Too close to call

Gemma 2 27B: 37.3 (#218), Phi-4: 36.9 (#226)

BenchmarkGemma 2 27BPhi-4
LMArena Longer Query12311217

Writing & Preference Gemma 2 27B leads

Gemma 2 27B: 44.2 (#225), Phi-4: 40.5 (#244)

BenchmarkGemma 2 27BPhi-4
LMArena Text12311217
LMArena Creative Writing12411182
LMArena Multi-Turn12241206
LiveBench Language32.6%25.6%
Short-Story Creative Writing—62.6%

Frequently asked questions

Is Gemma 2 27B better than Phi-4?

Phi-4 is the stronger model overall, scoring 31.2 to 29.4 on the Noometry Index.

Which is cheaper, Gemma 2 27B or Phi-4?

Phi-4 is cheaper. It lists at $0.07 per million input tokens and $0.14 per million output tokens; Gemma 2 27B lists at $0.65 and $0.65.

Is Gemma 2 27B or Phi-4 better for coding?

They score almost the same on coding (34.1 vs 34.4); test both on your own repository before choosing.

Which has the bigger context window?

Phi-4 does, with 128K tokens against 8K.

How many benchmarks do Gemma 2 27B and Phi-4 share?

32 benchmarks have published results for both models. Gemma 2 27B has 34 scored results on Noometry and Phi-4 has 37.

Related comparisons

Go deeper