Noometry

Model comparison

Nemotron 4 340b Instruct vs o1

o1 is the stronger model overall, scoring 40.9 to 35.9 on the Noometry Index.

Last verified . 17 shared benchmarks.

Nemotron 4 340b Instruct NVIDIA

35.9

Rank #223 Confirmed

o1 OpenAI

40.9

Rank #143 Confirmed

Summary

  • They share 17 benchmarks with published results for both. Nemotron 4 340b Instruct scores higher in 0 categories and o1 in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in long context, where o1 leads 50.3 to 37.1.
  • Nemotron 4 340b Instruct has downloadable open weights; the other is API-only.

Side by side

Nemotron 4 340b Instructo1
ProviderNVIDIAOpenAI
Noometry Index35.940.9
Released2024-06-142024-09-12
WeightsOpenProprietary
Context window—200K
Max output—100K
Input $ / M tokens—$15
Output $ / M tokens—$60
Results tracked1752

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o1 leads

Nemotron 4 340b Instruct: 35.2 (#228), o1: 46.1 (#70)

BenchmarkNemotron 4 340b Instructo1
LMArena Coding12091367
Aider Polyglot—61.7%
WeirdML—47.6%
LiveBench Coding—69.7%
CadEval—56%
HumanEval+—89%
MBPP+—80.2%

Agentic & Tool Use Not comparable

Nemotron 4 340b Instruct: —, o1: 24.6 (#117)

BenchmarkNemotron 4 340b Instructo1
Cybench—10%
METR Time Horizons—51.1%

Reasoning o1 leads

Nemotron 4 340b Instruct: 23.4 (#178), o1: 27.9 (#111)

BenchmarkNemotron 4 340b Instructo1
LMArena Hard Prompts12021371
SimpleBench—41.7%
ARC-AGI-1—30.7%
Chess Puzzles—15%
EnigmaEval—5.7%
LiveBench Reasoning—91.6%
DTBench—74.7%
LiveBench Data Analysis—65.5%
LMCA—22.3%
Epoch Capabilities Index—141.91
LiveBench—75.7%

Math o1 leads

Nemotron 4 340b Instruct: 34.3 (#195), o1: 36.1 (#175)

BenchmarkNemotron 4 340b Instructo1
LMArena Math12161388
FrontierMath (Tiers 1-3)—14.7%
OTIS Mock AIME 2024-2025—73.3%
LiveBench Math—80.3%
MATH Level 5—94.7%
FrontierMath (Feb 2025 set)—9.3%

Knowledge o1 leads

Nemotron 4 340b Instruct: 32.0 (#216), o1: 41.5 (#110)

BenchmarkNemotron 4 340b Instructo1
LMArena Expert11711361
GPQA Diamond—76.8%
Humanity's Last Exam—8%
SimpleQA Verified—41.1%
Confabulations—11.7%

Multimodal Not comparable

Nemotron 4 340b Instruct: —, o1: 34.2 (#93)

BenchmarkNemotron 4 340b Instructo1
LMArena Vision—1168
GeoBench—80%
VPCT—37%
SpatialViz-Bench—41.4%

Multilingual o1 leads

Nemotron 4 340b Instruct: 37.9 (#233), o1: 48.6 (#142)

BenchmarkNemotron 4 340b Instructo1
LMArena Non-English12061358
LMArena Chinese12141394
LMArena French11971344
LMArena German11921337
LMArena Japanese11281346
LMArena Korean11481396
LMArena Russian12191356
LMArena Spanish11951345

Instruction Following o1 leads

Nemotron 4 340b Instruct: 63.1 (#232), o1: 74.8 (#86)

BenchmarkNemotron 4 340b Instructo1
LMArena Instruction Following12031367
LiveBench Instruction Following—81.5%

Long Context o1 leads

Nemotron 4 340b Instruct: 37.1 (#221), o1: 50.3 (#9)

BenchmarkNemotron 4 340b Instructo1
LMArena Longer Query12251378
Fiction.LiveBench—83.3%

Writing & Preference o1 leads

Nemotron 4 340b Instruct: 42.7 (#233), o1: 55.6 (#144)

BenchmarkNemotron 4 340b Instructo1
LMArena Text12251366
LMArena Creative Writing12031348
LMArena Multi-Turn12141369
Short-Story Creative Writing—70.2%
LiveBench Language—65.4%

Frequently asked questions

Is Nemotron 4 340b Instruct better than o1?

o1 is the stronger model overall, scoring 40.9 to 35.9 on the Noometry Index.

Is Nemotron 4 340b Instruct or o1 better for coding?

o1 scores higher on coding benchmarks: 46.1 versus 35.2 in the Noometry coding category.

How many benchmarks do Nemotron 4 340b Instruct and o1 share?

17 benchmarks have published results for both models. Nemotron 4 340b Instruct has 17 scored results on Noometry and o1 has 52.

Related comparisons

Go deeper