Noometry

Model comparison

GPT-4 vs Olmo 3.1 32b Instruct

Olmo 3.1 32b Instruct is the stronger model overall, scoring 39.4 to 29.1 on the Noometry Index.

Last verified . 16 shared benchmarks.

GPT-4 OpenAI

29.1

Rank #316 Confirmed

Olmo 3.1 32b Instruct Allen Institute for AI (Ai2)

39.4

Rank #168 Confirmed

Summary

  • They share 16 benchmarks with published results for both. GPT-4 scores higher in 0 categories and Olmo 3.1 32b Instruct in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where Olmo 3.1 32b Instruct leads 36.3 to 10.8.
  • Olmo 3.1 32b Instruct has downloadable open weights; the other is API-only.

Side by side

GPT-4Olmo 3.1 32b Instruct
ProviderOpenAIAllen Institute for AI (Ai2)
Noometry Index29.139.4
Released2023-03-14—
WeightsProprietaryOpen
Context window8K—
Max output8K—
Input $ / M tokens$30—
Output $ / M tokens$60—
Results tracked3816

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Olmo 3.1 32b Instruct leads

GPT-4: 31.6 (#283), Olmo 3.1 32b Instruct: 39.5 (#157)

BenchmarkGPT-4Olmo 3.1 32b Instruct
LMArena Coding12541347
WeirdML12.4%—
BigCodeBench Instruct46%—
BigCodeBench Complete57.2%—
HumanEval+79.3%—

Agentic & Tool Use Not comparable

GPT-4: —, Olmo 3.1 32b Instruct: —

BenchmarkGPT-4Olmo 3.1 32b Instruct
METR Time Horizons36.1%—

Reasoning Olmo 3.1 32b Instruct leads

GPT-4: 17.8 (#289), Olmo 3.1 32b Instruct: 26.4 (#132)

BenchmarkGPT-4Olmo 3.1 32b Instruct
LMArena Hard Prompts12411322
Chess Puzzles4%—
Mystery Game Puzzles12%—
DTBench62.7%—
LMCA17.1%—
BIG-Bench Hard75.1%—
Epoch Capabilities Index125.89—
ForecastBench57.8—
HellaSwag95.3%—
WinoGrande87.5%—

Math Olmo 3.1 32b Instruct leads

GPT-4: 10.8 (#309), Olmo 3.1 32b Instruct: 36.3 (#167)

BenchmarkGPT-4Olmo 3.1 32b Instruct
LMArena Math12691305
OTIS Mock AIME 2024-20251.1%—
MATH Level 523%—
GSM8K92%—

Knowledge Olmo 3.1 32b Instruct leads

GPT-4: 18.4 (#282), Olmo 3.1 32b Instruct: 36.1 (#175)

BenchmarkGPT-4Olmo 3.1 32b Instruct
LMArena Expert12111308
GPQA Diamond35.7%—
MMLU86.4%—
TriviaQA84.8%—

Multilingual Olmo 3.1 32b Instruct leads

GPT-4: 40.6 (#215), Olmo 3.1 32b Instruct: 42.6 (#191)

BenchmarkGPT-4Olmo 3.1 32b Instruct
LMArena Non-English12461275
LMArena Chinese12421304
LMArena French12831328
LMArena German12511282
LMArena Korean11841206
LMArena Russian12511268
LMArena Spanish12611336
LMArena Japanese1209—

Instruction Following Olmo 3.1 32b Instruct leads

GPT-4: 65.3 (#222), Olmo 3.1 32b Instruct: 68.6 (#187)

BenchmarkGPT-4Olmo 3.1 32b Instruct
LMArena Instruction Following12411299

Long Context Olmo 3.1 32b Instruct leads

GPT-4: 37.7 (#212), Olmo 3.1 32b Instruct: 39.9 (#166)

BenchmarkGPT-4Olmo 3.1 32b Instruct
LMArena Longer Query12441312

Writing & Preference Olmo 3.1 32b Instruct leads

GPT-4: 34.9 (#268), Olmo 3.1 32b Instruct: 50.2 (#185)

BenchmarkGPT-4Olmo 3.1 32b Instruct
LMArena Text12631311
LMArena Creative Writing12441264
LMArena Multi-Turn12571309
EQ-Bench Creative Writing752—

Frequently asked questions

Is GPT-4 better than Olmo 3.1 32b Instruct?

Olmo 3.1 32b Instruct is the stronger model overall, scoring 39.4 to 29.1 on the Noometry Index.

Is GPT-4 or Olmo 3.1 32b Instruct better for coding?

Olmo 3.1 32b Instruct scores higher on coding benchmarks: 39.5 versus 31.6 in the Noometry coding category.

How many benchmarks do GPT-4 and Olmo 3.1 32b Instruct share?

16 benchmarks have published results for both models. GPT-4 has 38 scored results on Noometry and Olmo 3.1 32b Instruct has 16.

Related comparisons

Go deeper