Model comparison

Gemma 7B vs o1

o1 is the stronger model overall, scoring 40.9 to 30.0 on the Noometry Index.

Last verified October 11, 2026. 16 shared benchmarks.

Gemma 7B

Google

30.0

Rank #299 Confirmed

o1 OpenAI

40.9

Rank #143 Confirmed

Summary

They share 16 benchmarks with published results for both. Gemma 7B scores higher in 0 categories and o1 in 8 categories; 8 gaps are clear of the uncertainty.
The widest gap is in writing & preference, where o1 leads 55.6 to 27.1.
Gemma 7B has downloadable open weights; the other is API-only.

Side by side

Gemma 7B and o1 specifications
	Gemma 7B	o1
Provider	Google	OpenAI
Noometry Index	30.0	40.9
Released	2024-02-21	2024-09-12
Weights	Open	Proprietary
Context window	—	200K
Max output	—	100K
Input $ / M tokens	—	$15
Output $ / M tokens	—	$60
Results tracked	27	52

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o1 leads

Gemma 7B: 30.5 (#294), o1: 46.1 (#70)

Coding benchmarks
Benchmark	Gemma 7B	o1
LMArena Coding	1048	1367
HumanEval+	28.7%	89%
MBPP+	43.4%	80.2%
Aider Polyglot	—	61.7%
WeirdML	—	47.6%
LiveBench Coding	—	69.7%
CadEval	—	56%

Agentic & Tool Use Not comparable

Gemma 7B: —, o1: 24.6 (#117)

Agentic & Tool Use benchmarks
Benchmark	Gemma 7B	o1
Cybench	—	10%
METR Time Horizons	—	51.1%

Reasoning o1 leads

Gemma 7B: 19.9 (#249), o1: 27.9 (#111)

Reasoning benchmarks
Benchmark	Gemma 7B	o1
LMArena Hard Prompts	1042	1371
Epoch Capabilities Index	111.99	141.91
SimpleBench	—	41.7%
ARC-AGI-1	—	30.7%
Chess Puzzles	—	15%
EnigmaEval	—	5.7%
LiveBench Reasoning	—	91.6%
DTBench	—	74.7%
LiveBench Data Analysis	—	65.5%
LMCA	—	22.3%
Adversarial NLI	48.7%	—
BIG-Bench Hard	55.1%	—
HellaSwag	82.2%	—
LiveBench	—	75.7%
PIQA	81.2%	—
WinoGrande	79%	—

Math o1 leads

Gemma 7B: 31.2 (#228), o1: 36.1 (#175)

Math benchmarks
Benchmark	Gemma 7B	o1
LMArena Math	1066	1388
FrontierMath (Tiers 1-3)	—	14.7%
OTIS Mock AIME 2024-2025	—	73.3%
LiveBench Math	—	80.3%
MATH Level 5	—	94.7%
FrontierMath (Feb 2025 set)	—	9.3%
GSM8K	46.4%	—

Knowledge o1 leads

Gemma 7B: 27.3 (#252), o1: 41.5 (#110)

Knowledge benchmarks
Benchmark	Gemma 7B	o1
LMArena Expert	1001	1361
GPQA Diamond	—	76.8%
Humanity's Last Exam	—	8%
SimpleQA Verified	—	41.1%
Confabulations	—	11.7%
ARC (AI2) Challenge	78.3%	—
BoolQ	83.2%	—
MMLU	66.1%	—
OpenBookQA	78.6%	—
TriviaQA	72.3%	—

Multimodal Not comparable

Gemma 7B: —, o1: 34.2 (#93)

Multimodal benchmarks
Benchmark	Gemma 7B	o1
LMArena Vision	—	1168
GeoBench	—	80%
VPCT	—	37%
SpatialViz-Bench	—	41.4%

Multilingual o1 leads

Gemma 7B: 25.1 (#287), o1: 48.6 (#142)

Multilingual benchmarks
Benchmark	Gemma 7B	o1
LMArena Non-English	999	1358
LMArena Chinese	1035	1394
LMArena French	1025	1344
LMArena Russian	993	1356
LMArena German	—	1337
LMArena Japanese	—	1346
LMArena Korean	—	1396
LMArena Spanish	—	1345

Instruction Following o1 leads

Gemma 7B: 51.5 (#295), o1: 74.8 (#86)

Instruction Following benchmarks
Benchmark	Gemma 7B	o1
LMArena Instruction Following	1017	1367
LiveBench Instruction Following	—	81.5%

Long Context o1 leads

Gemma 7B: 31.1 (#282), o1: 50.3 (#9)

Long Context benchmarks
Benchmark	Gemma 7B	o1
LMArena Longer Query	1022	1378
Fiction.LiveBench	—	83.3%

Writing & Preference o1 leads

Gemma 7B: 27.1 (#302), o1: 55.6 (#144)

Writing & Preference benchmarks
Benchmark	Gemma 7B	o1
LMArena Text	1056	1366
LMArena Creative Writing	1024	1348
LMArena Multi-Turn	963	1369
Short-Story Creative Writing	—	70.2%
LiveBench Language	—	65.4%

Frequently asked questions

Is Gemma 7B better than o1?