Noometry

Model comparison

gpt-oss-20b vs Llama 3.1-8B

gpt-oss-20b is the stronger model overall, scoring 32.5 to 23.0 on the Noometry Index.

Last verified . 31 shared benchmarks.

gpt-oss-20b OpenAI

32.5

Rank #255 Confirmed

Llama 3.1-8B Meta

23.0

Rank #352 Confirmed

Summary

  • They share 31 benchmarks with published results for both. gpt-oss-20b scores higher in 8 categories and Llama 3.1-8B in 1 category; 9 gaps are clear of the uncertainty.
  • The widest gap is in math, where gpt-oss-20b leads 39.4 to 10.2.
  • The biggest single-benchmark swing is OTIS Mock AIME 2024-2025: 65.3% for gpt-oss-20b and 1.7% for Llama 3.1-8B.
  • gpt-oss-20b is cheaper at $0.018 / $0.09 per million input/output tokens, against $0.05 / $0.08 for Llama 3.1-8B.
  • gpt-oss-20b accepts more context: 131K tokens versus 128K.

Side by side

gpt-oss-20bLlama 3.1-8B
ProviderOpenAIMeta
Noometry Index32.523.0
Released2025-08-052024-07-23
WeightsOpenOpen
Context window131K128K
Max output16K4K
Input $ / M tokens$0.018$0.05
Output $ / M tokens$0.09$0.08
Results tracked3443

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding gpt-oss-20b leads

gpt-oss-20b: 37.6 (#192), Llama 3.1-8B: 20.2 (#340)

Benchmarkgpt-oss-20bLlama 3.1-8B
SciCode34.4%13.2%
WeirdML40.9%1.7%
LMArena Coding13061195
BigCodeBench Instruct—32.8%
BigCodeBench Complete—40.5%
ALE-Bench566.05—
HumanEval+—62.8%
MBPP+—55.6%

Agentic & Tool Use Llama 3.1-8B leads

gpt-oss-20b: 9.3 (#154), Llama 3.1-8B: 22.5 (#131)

Benchmarkgpt-oss-20bLlama 3.1-8B
Terminal-Bench3.4%—
Berkeley Function Calling Leaderboard—25.8%
BALROG—15.1%

Reasoning gpt-oss-20b leads

gpt-oss-20b: 19.3 (#261), Llama 3.1-8B: 14.9 (#321)

Benchmarkgpt-oss-20bLlama 3.1-8B
CritPt1.4%0%
Chess Puzzles4%0%
LMArena Hard Prompts12741175
DTBench68%50.9%
LMCA14.5%5.4%
Epoch Capabilities Index137.82116.57
Kagi LLM Benchmark53.2%—
PIQA—81.2%

Math gpt-oss-20b leads

gpt-oss-20b: 39.4 (#103), Llama 3.1-8B: 10.2 (#317)

Benchmarkgpt-oss-20bLlama 3.1-8B
OTIS Mock AIME 2024-202565.3%1.7%
Omni-MATH56.5%13.7%
LMArena Math13171179
MATH Level 5—22.9%
GSM8K—82.4%

Knowledge gpt-oss-20b leads

gpt-oss-20b: 34.6 (#195), Llama 3.1-8B: 8.0 (#307)

Benchmarkgpt-oss-20bLlama 3.1-8B
GPQA Diamond60.8%27%
MMLU-Pro74%40.6%
GPQA (HELM)59.4%24.7%
LMArena Expert12581144
BoolQ—82.8%
MMLU—56.1%

Multilingual gpt-oss-20b leads

gpt-oss-20b: 42.2 (#197), Llama 3.1-8B: 34.0 (#249)

Benchmarkgpt-oss-20bLlama 3.1-8B
LMArena Non-English12681148
LMArena Chinese13141151
LMArena German12551144
LMArena Japanese12441061
LMArena Korean12361053
LMArena Russian12781158
LMArena Spanish12671169
LMArena French—1177

Instruction Following gpt-oss-20b leads

gpt-oss-20b: 61.8 (#240), Llama 3.1-8B: 58.9 (#258)

Benchmarkgpt-oss-20bLlama 3.1-8B
IFEval73.2%74.3%
LMArena Instruction Following12361159

Long Context gpt-oss-20b leads

gpt-oss-20b: 37.9 (#209), Llama 3.1-8B: 35.8 (#238)

Benchmarkgpt-oss-20bLlama 3.1-8B
LMArena Longer Query12501182

Writing & Preference gpt-oss-20b leads

gpt-oss-20b: 35.5 (#265), Llama 3.1-8B: 29.7 (#290)

Benchmarkgpt-oss-20bLlama 3.1-8B
LMArena Text12871187
LMArena Creative Writing12011154
EQ-Bench Creative Writing666713
WildBench73.7%68.7%
LMArena Multi-Turn12681172

Frequently asked questions

Is gpt-oss-20b better than Llama 3.1-8B?

gpt-oss-20b is the stronger model overall, scoring 32.5 to 23.0 on the Noometry Index.

Which is cheaper, gpt-oss-20b or Llama 3.1-8B?

gpt-oss-20b is cheaper. It lists at $0.018 per million input tokens and $0.09 per million output tokens; Llama 3.1-8B lists at $0.05 and $0.08.

Is gpt-oss-20b or Llama 3.1-8B better for coding?

gpt-oss-20b scores higher on coding benchmarks: 37.6 versus 20.2 in the Noometry coding category.

Which has the bigger context window?

gpt-oss-20b does, with 131K tokens against 128K.

How many benchmarks do gpt-oss-20b and Llama 3.1-8B share?

31 benchmarks have published results for both models. gpt-oss-20b has 34 scored results on Noometry and Llama 3.1-8B has 43.

Related comparisons

Go deeper