Noometry

Model comparison

Gemma 2 27B vs Step 3.5 Flash

Step 3.5 Flash is the stronger model overall, scoring 42.3 to 29.4 on the Noometry Index.

Last verified . 17 shared benchmarks.

Gemma 2 27B Google

29.4

Rank #312 Confirmed

Step 3.5 Flash StepFun

42.3

Rank #116 Confirmed

Summary

  • They share 17 benchmarks with published results for both. Gemma 2 27B scores higher in 0 categories and Step 3.5 Flash in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where Step 3.5 Flash leads 42.6 to 10.7.
  • Step 3.5 Flash is cheaper at $0.10 / $0.30 per million input/output tokens, against $0.65 / $0.65 for Gemma 2 27B.
  • Step 3.5 Flash accepts more context: 256K tokens versus 8K.

Side by side

Gemma 2 27BStep 3.5 Flash
ProviderGoogleStepFun
Noometry Index29.442.3
Released2024-06-242026-01-29
WeightsOpenOpen
Context window8K256K
Max output2K256K
Input $ / M tokens$0.65$0.10
Output $ / M tokens$0.65$0.30
Results tracked3419

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Step 3.5 Flash leads

Gemma 2 27B: 34.1 (#246), Step 3.5 Flash: 42.4 (#105)

BenchmarkGemma 2 27BStep 3.5 Flash
LMArena Coding12111436
BigCodeBench Instruct42.8%—
LiveBench Coding36%—
BigCodeBench Complete52.5%—

Reasoning Step 3.5 Flash leads

Gemma 2 27B: 15.3 (#315), Step 3.5 Flash: 22.2 (#202)

BenchmarkGemma 2 27BStep 3.5 Flash
LMArena Hard Prompts11981411
NYT Connections (extended)—28.4%
LiveBench Reasoning28.1%—
DTBench48%—
LiveBench Data Analysis47.9%—
LMCA7.1%—
Epoch Capabilities Index122.08—
LiveBench38.2%—

Math Step 3.5 Flash leads

Gemma 2 27B: 10.7 (#311), Step 3.5 Flash: 42.6 (#84)

BenchmarkGemma 2 27BStep 3.5 Flash
LMArena Math12121408
MathArena Final-Answer Competitions—66.8%
OTIS Mock AIME 2024-20251.4%—
LiveBench Math26.5%—
MATH Level 527.9%—

Knowledge Step 3.5 Flash leads

Gemma 2 27B: 19.0 (#280), Step 3.5 Flash: 39.6 (#132)

BenchmarkGemma 2 27BStep 3.5 Flash
LMArena Expert11721421
GPQA Diamond36.5%—
Confabulations27.1%—
MMLU75.7%—

Multilingual Step 3.5 Flash leads

Gemma 2 27B: 38.6 (#226), Step 3.5 Flash: 50.5 (#119)

BenchmarkGemma 2 27BStep 3.5 Flash
LMArena Non-English12171385
LMArena Chinese12211447
LMArena French12471421
LMArena German12091405
LMArena Japanese11751354
LMArena Korean11741352
LMArena Russian12341385
LMArena Spanish12281419

Instruction Following Step 3.5 Flash leads

Gemma 2 27B: 60.5 (#249), Step 3.5 Flash: 73.1 (#124)

BenchmarkGemma 2 27BStep 3.5 Flash
LMArena Instruction Following12061385
LiveBench Instruction Following58.1%—

Long Context Step 3.5 Flash leads

Gemma 2 27B: 37.3 (#218), Step 3.5 Flash: 42.8 (#117)

BenchmarkGemma 2 27BStep 3.5 Flash
LMArena Longer Query12311402

Writing & Preference Step 3.5 Flash leads

Gemma 2 27B: 44.2 (#225), Step 3.5 Flash: 58.8 (#113)

BenchmarkGemma 2 27BStep 3.5 Flash
LMArena Text12311403
LMArena Creative Writing12411357
LMArena Multi-Turn12241405
LiveBench Language32.6%—

Frequently asked questions

Is Gemma 2 27B better than Step 3.5 Flash?

Step 3.5 Flash is the stronger model overall, scoring 42.3 to 29.4 on the Noometry Index.

Which is cheaper, Gemma 2 27B or Step 3.5 Flash?

Step 3.5 Flash is cheaper. It lists at $0.10 per million input tokens and $0.30 per million output tokens; Gemma 2 27B lists at $0.65 and $0.65.

Is Gemma 2 27B or Step 3.5 Flash better for coding?

Step 3.5 Flash scores higher on coding benchmarks: 42.4 versus 34.1 in the Noometry coding category.

Which has the bigger context window?

Step 3.5 Flash does, with 256K tokens against 8K.

How many benchmarks do Gemma 2 27B and Step 3.5 Flash share?

17 benchmarks have published results for both models. Gemma 2 27B has 34 scored results on Noometry and Step 3.5 Flash has 19.

Related comparisons

Go deeper