Noometry

Model comparison

Gemma 3 1B vs Grok 4.20 (Non-Reasoning)

Grok 4.20 (Non-Reasoning) is the stronger model overall, scoring 48.6 to 21.1 on the Noometry Index.

Last verified . 3 shared benchmarks.

Gemma 3 1B Google

21.1

Rank #353 Reported

Grok 4.20 (Non-Reasoning) xAI

48.6

Rank #54 Confirmed

Summary

  • They share 3 benchmarks with published results for both. Gemma 3 1B scores higher in 0 categories and Grok 4.20 (Non-Reasoning) in 4 categories; 4 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Grok 4.20 (Non-Reasoning) leads 52.8 to 7.0.
  • The biggest single-benchmark swing is OTIS Mock AIME 2024-2025: 1.1% for Gemma 3 1B and 92.2% for Grok 4.20 (Non-Reasoning).
  • Gemma 3 1B has downloadable open weights; the other is API-only.

Side by side

Gemma 3 1BGrok 4.20 (Non-Reasoning)
ProviderGooglexAI
Noometry Index21.148.6
Released2025-03-122026-02-17
WeightsOpenProprietary
Context window—1M
Max output—30K
Input $ / M tokens—$1.25
Output $ / M tokens—$2.50
Results tracked446

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Gemma 3 1B: —, Grok 4.20 (Non-Reasoning): 42.1 (#112)

BenchmarkGemma 3 1BGrok 4.20 (Non-Reasoning)
LMArena WebDev—1375
WeirdML—52.3%
LMArena Coding—1459
ALE-Bench—1,150

Agentic & Tool Use Grok 4.20 (Non-Reasoning) leads

Gemma 3 1B: 13.7 (#152), Grok 4.20 (Non-Reasoning): 34.4 (#46)

BenchmarkGemma 3 1BGrok 4.20 (Non-Reasoning)
Terminal-Bench—57.3%
Berkeley Function Calling Leaderboard7.2%—
τ²-bench Banking—18%
LMArena Search—1189
Vending-Bench 2—4,663

Reasoning Grok 4.20 (Non-Reasoning) leads

Gemma 3 1B: 19.2 (#264), Grok 4.20 (Non-Reasoning): 52.3 (#32)

BenchmarkGemma 3 1BGrok 4.20 (Non-Reasoning)
Chess Puzzles0%24%
ARC-AGI-2—65.1%
Kagi LLM Benchmark—75%
NYT Connections (extended)—85.4%
ARC-AGI-1—89.5%
Thematic Generalization—63.8%
LMArena Hard Prompts—1451
DTBench—90.1%
LMCA—38.7%
Epoch Capabilities Index—151.98
ForecastBench—61.4

Math Grok 4.20 (Non-Reasoning) leads

Gemma 3 1B: 10.2 (#316), Grok 4.20 (Non-Reasoning): 48.2 (#65)

BenchmarkGemma 3 1BGrok 4.20 (Non-Reasoning)
OTIS Mock AIME 2024-20251.1%92.2%
FrontierMath (Tiers 1-3)—44.9%
FrontierMath Tier 4—17.1%
ProofBench—14%
LMArena Math—1455

Knowledge Grok 4.20 (Non-Reasoning) leads

Gemma 3 1B: 7.0 (#314), Grok 4.20 (Non-Reasoning): 52.8 (#60)

BenchmarkGemma 3 1BGrok 4.20 (Non-Reasoning)
GPQA Diamond19.9%89.3%
SimpleQA Verified—30.2%
LMArena Expert—1439

Multimodal Not comparable

Gemma 3 1B: —, Grok 4.20 (Non-Reasoning): 33.3 (#98)

BenchmarkGemma 3 1BGrok 4.20 (Non-Reasoning)
LMArena Vision—1263
Blueprint-Bench 2—0%
LMArena Document—1416

Multilingual Not comparable

Gemma 3 1B: —, Grok 4.20 (Non-Reasoning): 54.5 (#40)

BenchmarkGemma 3 1BGrok 4.20 (Non-Reasoning)
LMArena Non-English—1441
LMArena Chinese—1481
LMArena French—1476
LMArena German—1465
LMArena Japanese—1449
LMArena Korean—1417
LMArena Russian—1458
LMArena Spanish—1443

Instruction Following Not comparable

Gemma 3 1B: —, Grok 4.20 (Non-Reasoning): 74.8 (#83)

BenchmarkGemma 3 1BGrok 4.20 (Non-Reasoning)
LMArena Instruction Following—1420

Long Context Not comparable

Gemma 3 1B: —, Grok 4.20 (Non-Reasoning): 45.5 (#34)

BenchmarkGemma 3 1BGrok 4.20 (Non-Reasoning)
CL-bench—22.2%
CL-bench Life—11.9%
LMArena Longer Query—1437

Writing & Preference Not comparable

Gemma 3 1B: —, Grok 4.20 (Non-Reasoning): 65.7 (#44)

BenchmarkGemma 3 1BGrok 4.20 (Non-Reasoning)
LMArena Text—1451
LMArena Creative Writing—1438
EQ-Bench Creative Writing—1574
LMArena Multi-Turn—1456

Frequently asked questions

Is Gemma 3 1B better than Grok 4.20 (Non-Reasoning)?

Grok 4.20 (Non-Reasoning) is the stronger model overall, scoring 48.6 to 21.1 on the Noometry Index.

How many benchmarks do Gemma 3 1B and Grok 4.20 (Non-Reasoning) share?

3 benchmarks have published results for both models. Gemma 3 1B has 4 scored results on Noometry and Grok 4.20 (Non-Reasoning) has 46.

Related comparisons

Go deeper