Noometry

Model comparison

GPT-4 Turbo vs Nemotron 4 340b Instruct

Nemotron 4 340b Instruct is the stronger model overall, scoring 35.9 to 30.5 on the Noometry Index.

Last verified . 17 shared benchmarks.

GPT-4 Turbo OpenAI

30.5

Rank #292 Confirmed

Nemotron 4 340b Instruct NVIDIA

35.9

Rank #223 Confirmed

Summary

  • They share 17 benchmarks with published results for both. GPT-4 Turbo scores higher in 4 categories and Nemotron 4 340b Instruct in 4 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in math, where Nemotron 4 340b Instruct leads 34.3 to 9.0.
  • Nemotron 4 340b Instruct has downloadable open weights; the other is API-only.

Side by side

GPT-4 TurboNemotron 4 340b Instruct
ProviderOpenAINVIDIA
Noometry Index30.535.9
Released2023-11-062024-06-14
WeightsProprietaryOpen
Context window128K—
Max output4K—
Input $ / M tokens$10—
Output $ / M tokens$30—
Results tracked3617

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Nemotron 4 340b Instruct leads

GPT-4 Turbo: 33.8 (#249), Nemotron 4 340b Instruct: 35.2 (#228)

BenchmarkGPT-4 TurboNemotron 4 340b Instruct
LMArena Coding12681209
WeirdML18%—
BigCodeBench Instruct48.2%—
BigCodeBench Complete58.2%—
HumanEval+86.6%—
MBPP+73.3%—

Agentic & Tool Use Not comparable

GPT-4 Turbo: —, Nemotron 4 340b Instruct: —

BenchmarkGPT-4 TurboNemotron 4 340b Instruct
METR Time Horizons36.7%—

Reasoning Nemotron 4 340b Instruct leads

GPT-4 Turbo: 15.3 (#317), Nemotron 4 340b Instruct: 23.4 (#178)

BenchmarkGPT-4 TurboNemotron 4 340b Instruct
LMArena Hard Prompts12511202
SimpleBench25.1%—
Chess Puzzles6%—
DTBench61.6%—
LMCA9.8%—
Epoch Capabilities Index127.25—
ForecastBench59.4—

Math Nemotron 4 340b Instruct leads

GPT-4 Turbo: 9.0 (#322), Nemotron 4 340b Instruct: 34.3 (#195)

BenchmarkGPT-4 TurboNemotron 4 340b Instruct
LMArena Math12721216
FrontierMath (Tiers 1-3)0.7%—
OTIS Mock AIME 2024-20256.7%—
MATH Level 546.7%—

Knowledge Nemotron 4 340b Instruct leads

GPT-4 Turbo: 24.3 (#268), Nemotron 4 340b Instruct: 32.0 (#216)

BenchmarkGPT-4 TurboNemotron 4 340b Instruct
LMArena Expert12231171
GPQA Diamond46.6%—
Confabulations28.4%—
MMLU81.3%—

Multimodal Not comparable

GPT-4 Turbo: 30.6 (#110), Nemotron 4 340b Instruct: —

BenchmarkGPT-4 TurboNemotron 4 340b Instruct
LMArena Vision1090—

Multilingual GPT-4 Turbo leads

GPT-4 Turbo: 40.5 (#216), Nemotron 4 340b Instruct: 37.9 (#233)

BenchmarkGPT-4 TurboNemotron 4 340b Instruct
LMArena Non-English12451206
LMArena Chinese12421214
LMArena French12761197
LMArena German12591192
LMArena Japanese11941128
LMArena Korean11871148
LMArena Russian12591219
LMArena Spanish12601195

Instruction Following GPT-4 Turbo leads

GPT-4 Turbo: 65.8 (#216), Nemotron 4 340b Instruct: 63.1 (#232)

BenchmarkGPT-4 TurboNemotron 4 340b Instruct
LMArena Instruction Following12491203

Long Context Too close to call

GPT-4 Turbo: 38.0 (#206), Nemotron 4 340b Instruct: 37.1 (#221)

BenchmarkGPT-4 TurboNemotron 4 340b Instruct
LMArena Longer Query12541225

Writing & Preference GPT-4 Turbo leads

GPT-4 Turbo: 47.7 (#206), Nemotron 4 340b Instruct: 42.7 (#233)

BenchmarkGPT-4 TurboNemotron 4 340b Instruct
LMArena Text12721225
LMArena Creative Writing12691203
LMArena Multi-Turn12671214

Frequently asked questions

Is GPT-4 Turbo better than Nemotron 4 340b Instruct?

Nemotron 4 340b Instruct is the stronger model overall, scoring 35.9 to 30.5 on the Noometry Index.

Is GPT-4 Turbo or Nemotron 4 340b Instruct better for coding?

Nemotron 4 340b Instruct scores higher on coding benchmarks: 35.2 versus 33.8 in the Noometry coding category.

How many benchmarks do GPT-4 Turbo and Nemotron 4 340b Instruct share?

17 benchmarks have published results for both models. GPT-4 Turbo has 36 scored results on Noometry and Nemotron 4 340b Instruct has 17.

Related comparisons

Go deeper