Noometry

Model comparison

Claude 2 vs GPT-4o mini

Claude 2 and GPT-4o mini score almost the same on the Noometry Index (25.0 vs 25.5), so choose on price, context window or the category you care about most.

Last verified . 7 shared benchmarks.

Claude 2 Anthropic

25.0

Rank #346 Reported

GPT-4o mini OpenAI

25.5

Rank #343 Confirmed

Summary

  • They share 7 benchmarks with published results for both. Claude 2 scores higher in 1 category and GPT-4o mini in 2 categories; 2 gaps are clear of the uncertainty.
  • The widest gap is in reasoning, where Claude 2 leads 21.7 to 8.7.
  • The biggest single-benchmark swing is MATH Level 5: 11.7% for Claude 2 and 52.6% for GPT-4o mini.

Side by side

Claude 2GPT-4o mini
ProviderAnthropicOpenAI
Noometry Index25.025.5
Released2023-07-112024-07-18
WeightsProprietaryProprietary
Context window—128K
Max output—16K
Input $ / M tokens—$0.15
Output $ / M tokens—$0.60
Results tracked860

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Claude 2: —, GPT-4o mini: 22.0 (#335)

BenchmarkClaude 2GPT-4o mini
HumanEval+61.6%83.5%
Aider Polyglot—3.6%
WeirdML—11.8%
BigCodeBench Instruct—46.1%
LiveBench Coding—43.1%
LMArena Coding—1290
BigCodeBench Complete—57.4%
MBPP+—72.2%

Agentic & Tool Use Not comparable

Claude 2: —, GPT-4o mini: 27.5 (#101)

BenchmarkClaude 2GPT-4o mini
BALROG—17.4%

Reasoning Claude 2 leads

Claude 2: 21.7 (#216), GPT-4o mini: 8.7 (#347)

BenchmarkClaude 2GPT-4o mini
DTBench51.9%54.4%
Epoch Capabilities Index120.13126.56
ARC-AGI-2—0%
SimpleBench—10.7%
Kagi LLM Benchmark—28.8%
Chess Puzzles—0%
LiveBench Reasoning—32.8%
LMArena Hard Prompts—1267
Mystery Game Puzzles—12%
LiveBench Data Analysis—50%
LMCA—10.4%
LiveBench—41.3%
PIQA—88.7%

Math GPT-4o mini leads

Claude 2: 9.3 (#320), GPT-4o mini: 10.4 (#314)

BenchmarkClaude 2GPT-4o mini
OTIS Mock AIME 2024-20252.5%6.9%
MATH Level 511.7%52.6%
FrontierMath (Tiers 1-3)—0.7%
Omni-MATH—28%
LiveBench Math—36.3%
LMArena Math—1267
GSM8K—91.3%

Knowledge Too close to call

Claude 2: 16.9 (#287), GPT-4o mini: 17.7 (#284)

BenchmarkClaude 2GPT-4o mini
GPQA Diamond34.7%37.7%
MMLU78.5%81.8%
SimpleQA Verified—8.3%
MMLU-Pro—60.3%
Confabulations—37.2%
GPQA (HELM)—36.8%
LMArena Expert—1235
BoolQ—88.7%
TriviaQA87.5%—

Multimodal Not comparable

Claude 2: —, GPT-4o mini: 25.9 (#122)

BenchmarkClaude 2GPT-4o mini
LMArena Vision—1066
Video-MME—64.8%
GeoBench—64%
VPCT—34%

Multilingual Not comparable

Claude 2: —, GPT-4o mini: 42.0 (#199)

BenchmarkClaude 2GPT-4o mini
LMArena Non-English—1266
LMArena Chinese—1265
LMArena French—1297
LMArena German—1272
LMArena Japanese—1216
LMArena Korean—1195
LMArena Russian—1275
LMArena Spanish—1276

Instruction Following Not comparable

Claude 2: —, GPT-4o mini: 61.9 (#239)

BenchmarkClaude 2GPT-4o mini
LiveBench Instruction Following—56.8%
IFEval—78.2%
LMArena Instruction Following—1258

Long Context Not comparable

Claude 2: —, GPT-4o mini: 39.1 (#186)

BenchmarkClaude 2GPT-4o mini
LMArena Longer Query—1289

Writing & Preference Not comparable

Claude 2: —, GPT-4o mini: 39.5 (#248)

BenchmarkClaude 2GPT-4o mini
LMArena Text—1286
LMArena Creative Writing—1268
Short-Story Creative Writing—67.2%
EQ-Bench Creative Writing—873
WildBench—79.1%
LMArena Multi-Turn—1285
LiveBench Language—28.6%

Frequently asked questions

Is Claude 2 better than GPT-4o mini?

Claude 2 and GPT-4o mini score almost the same on the Noometry Index (25.0 vs 25.5), so choose on price, context window or the category you care about most.

How many benchmarks do Claude 2 and GPT-4o mini share?

7 benchmarks have published results for both models. Claude 2 has 8 scored results on Noometry and GPT-4o mini has 60.

Related comparisons

Go deeper