Noometry

Model comparison

Gemma 2 27B vs OLMo 2 Furious 13B

Gemma 2 27B and OLMo 2 Furious 13B score almost the same on the Noometry Index (29.2 vs 29.2), so choose on price, context window or the category you care about most.

Last verified . 7 shared benchmarks.

Gemma 2 27B Google

29.2

Rank #302 Confirmed

OLMo 2 Furious 13B Allen Institute for AI (Ai2)

29.2

Rank #301 Confirmed

Summary

  • They share 7 benchmarks with published results for both. Gemma 2 27B scores higher in 3 categories and OLMo 2 Furious 13B in 3 categories; 3 gaps are clear of the uncertainty.
  • The widest gap is in math, where OLMo 2 Furious 13B leads 21.3 to 10.0.
  • The biggest single-benchmark swing is LiveBench Data Analysis: 47.9% for Gemma 2 27B and 20.6% for OLMo 2 Furious 13B.

Side by side

Gemma 2 27BOLMo 2 Furious 13B
ProviderGoogleAllen Institute for AI (Ai2)
Noometry Index29.229.2
Released2024-06-242024-12-31
WeightsOpenOpen
Context window8K—
Max output2K—
Input $ / M tokens$0.65—
Output $ / M tokens$0.65—
Results tracked3412

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Gemma 2 27B leads

Gemma 2 27B: 33.5 (#243), OLMo 2 Furious 13B: 28.1 (#309)

BenchmarkGemma 2 27BOLMo 2 Furious 13B
LiveBench Coding36%10.4%
BigCodeBench Instruct42.8%—
LMArena Coding1211—
BigCodeBench Complete52.5%—

Reasoning Too close to call

Gemma 2 27B: 14.8 (#313), OLMo 2 Furious 13B: 14.8 (#312)

BenchmarkGemma 2 27BOLMo 2 Furious 13B
LiveBench Reasoning28.1%16.3%
LiveBench Data Analysis47.9%20.6%
LiveBench38.2%22.1%
LMArena Hard Prompts1199—
DTBench48%—
LMCA7.1%—
Epoch Capabilities Index122.06—

Math OLMo 2 Furious 13B leads

Gemma 2 27B: 10.0 (#306), OLMo 2 Furious 13B: 21.3 (#270)

BenchmarkGemma 2 27BOLMo 2 Furious 13B
LiveBench Math26.5%13.6%
OTIS Mock AIME 2024-20251.4%—
Omni-MATH—15.6%
LMArena Math1212—
MATH Level 527.9%—

Knowledge Too close to call

Gemma 2 27B: 18.9 (#275), OLMo 2 Furious 13B: 18.0 (#278)

BenchmarkGemma 2 27BOLMo 2 Furious 13B
GPQA Diamond36.5%—
MMLU-Pro—31%
Confabulations27.1%—
GPQA (HELM)—31.6%
LMArena Expert1172—
MMLU75.7%—

Multilingual Not comparable

Gemma 2 27B: 38.8 (#221), OLMo 2 Furious 13B: —

BenchmarkGemma 2 27BOLMo 2 Furious 13B
LMArena Non-English1217—
LMArena Chinese1221—
LMArena French1246—
LMArena German1209—
LMArena Japanese1175—
LMArena Korean1174—
LMArena Russian1234—
LMArena Spanish1228—

Instruction Following Too close to call

Gemma 2 27B: 60.7 (#244), OLMo 2 Furious 13B: 61.1 (#243)

BenchmarkGemma 2 27BOLMo 2 Furious 13B
LiveBench Instruction Following58.1%60.6%
IFEval—73%
LMArena Instruction Following1206—

Long Context Not comparable

Gemma 2 27B: 37.3 (#213), OLMo 2 Furious 13B: —

BenchmarkGemma 2 27BOLMo 2 Furious 13B
LMArena Longer Query1231—

Writing & Preference Gemma 2 27B leads

Gemma 2 27B: 44.4 (#220), OLMo 2 Furious 13B: 43.0 (#226)

BenchmarkGemma 2 27BOLMo 2 Furious 13B
LiveBench Language32.6%11.2%
LMArena Text1231—
LMArena Creative Writing1241—
WildBench—68.9%
LMArena Multi-Turn1224—

Frequently asked questions

Is Gemma 2 27B better than OLMo 2 Furious 13B?

Gemma 2 27B and OLMo 2 Furious 13B score almost the same on the Noometry Index (29.2 vs 29.2), so choose on price, context window or the category you care about most.

Is Gemma 2 27B or OLMo 2 Furious 13B better for coding?

Gemma 2 27B scores higher on coding benchmarks: 33.5 versus 28.1 in the Noometry coding category.

How many benchmarks do Gemma 2 27B and OLMo 2 Furious 13B share?

7 benchmarks have published results for both models. Gemma 2 27B has 34 scored results on Noometry and OLMo 2 Furious 13B has 12.

Related comparisons

Go deeper