Noometry

Model comparison

gpt-oss-20b vs Grok-2 (Dec 2024)

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.7 to 32.5 on the Noometry Index.

Last verified . 21 shared benchmarks.

gpt-oss-20b OpenAI

32.5

Rank #255 Confirmed

Grok-2 (Dec 2024) xAI

33.7

Rank #239 Confirmed

Summary

  • They share 21 benchmarks with published results for both. gpt-oss-20b scores higher in 4 categories and Grok-2 (Dec 2024) in 4 categories; 6 gaps are clear of the uncertainty.
  • The widest gap is in math, where gpt-oss-20b leads 39.4 to 20.8.
  • The biggest single-benchmark swing is OTIS Mock AIME 2024-2025: 65.3% for gpt-oss-20b and 11.5% for Grok-2 (Dec 2024).
  • gpt-oss-20b has downloadable open weights; the other is API-only.

Side by side

gpt-oss-20bGrok-2 (Dec 2024)
ProviderOpenAIxAI
Noometry Index32.533.7
Released2025-08-052024-08-13
WeightsOpenProprietary
Context window131K—
Max output16K—
Input $ / M tokens$0.018—
Output $ / M tokens$0.09—
Results tracked3434

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding gpt-oss-20b leads

gpt-oss-20b: 37.6 (#192), Grok-2 (Dec 2024): 33.3 (#258)

Benchmarkgpt-oss-20bGrok-2 (Dec 2024)
WeirdML40.9%22.2%
LMArena Coding13061287
SciCode34.4%—
LiveBench Coding—46.4%
ALE-Bench566.05—

Agentic & Tool Use Not comparable

gpt-oss-20b: 9.3 (#154), Grok-2 (Dec 2024): —

Benchmarkgpt-oss-20bGrok-2 (Dec 2024)
Terminal-Bench3.4%—

Reasoning gpt-oss-20b leads

gpt-oss-20b: 19.3 (#261), Grok-2 (Dec 2024): 16.9 (#299)

Benchmarkgpt-oss-20bGrok-2 (Dec 2024)
LMArena Hard Prompts12741272
DTBench68%65.2%
Epoch Capabilities Index137.82130.48
SimpleBench—22.7%
Kagi LLM Benchmark53.2%—
CritPt1.4%—
Chess Puzzles4%—
LiveBench Reasoning—54.8%
LiveBench Data Analysis—54.5%
LMCA14.5%—
LiveBench—54.3%

Math gpt-oss-20b leads

gpt-oss-20b: 39.4 (#103), Grok-2 (Dec 2024): 20.8 (#284)

Benchmarkgpt-oss-20bGrok-2 (Dec 2024)
OTIS Mock AIME 2024-202565.3%11.5%
LMArena Math13171283
Omni-MATH56.5%—
LiveBench Math—54.9%
MATH Level 5—63.5%
FrontierMath (Feb 2025 set)—0.7%

Knowledge gpt-oss-20b leads

gpt-oss-20b: 34.6 (#195), Grok-2 (Dec 2024): 29.8 (#233)

Benchmarkgpt-oss-20bGrok-2 (Dec 2024)
GPQA Diamond60.8%53.8%
LMArena Expert12581254
MMLU-Pro74%—
Confabulations—20.1%
GPQA (HELM)59.4%—

Multilingual Too close to call

gpt-oss-20b: 42.2 (#197), Grok-2 (Dec 2024): 43.1 (#188)

Benchmarkgpt-oss-20bGrok-2 (Dec 2024)
LMArena Non-English12681282
LMArena Chinese13141289
LMArena German12551287
LMArena Japanese12441244
LMArena Korean12361237
LMArena Russian12781286
LMArena Spanish12671281
LMArena French—1318

Instruction Following Grok-2 (Dec 2024) leads

gpt-oss-20b: 61.8 (#240), Grok-2 (Dec 2024): 66.9 (#202)

Benchmarkgpt-oss-20bGrok-2 (Dec 2024)
LMArena Instruction Following12361270
LiveBench Instruction Following—69.6%
IFEval73.2%—

Long Context Too close to call

gpt-oss-20b: 37.9 (#209), Grok-2 (Dec 2024): 38.8 (#190)

Benchmarkgpt-oss-20bGrok-2 (Dec 2024)
LMArena Longer Query12501276

Writing & Preference Grok-2 (Dec 2024) leads

gpt-oss-20b: 35.5 (#265), Grok-2 (Dec 2024): 48.6 (#198)

Benchmarkgpt-oss-20bGrok-2 (Dec 2024)
LMArena Text12871305
LMArena Creative Writing12011284
LMArena Multi-Turn12681290
Short-Story Creative Writing—63.6%
EQ-Bench Creative Writing666—
WildBench73.7%—
LiveBench Language—45.6%

Frequently asked questions

Is gpt-oss-20b better than Grok-2 (Dec 2024)?

Grok-2 (Dec 2024) is the stronger model overall, scoring 33.7 to 32.5 on the Noometry Index.

Is gpt-oss-20b or Grok-2 (Dec 2024) better for coding?

gpt-oss-20b scores higher on coding benchmarks: 37.6 versus 33.3 in the Noometry coding category.

How many benchmarks do gpt-oss-20b and Grok-2 (Dec 2024) share?

21 benchmarks have published results for both models. gpt-oss-20b has 34 scored results on Noometry and Grok-2 (Dec 2024) has 34.

Related comparisons

Go deeper