Noometry

Model comparison

Llama 3.1 Nemotron Ultra 253b v1 vs Qwen1.5-110B

Llama 3.1 Nemotron Ultra 253b v1 is the stronger model overall, scoring 36.7 to 34.2 on the Noometry Index.

Last verified . 10 shared benchmarks.

Llama 3.1 Nemotron Ultra 253b v1 NVIDIA

36.7

Rank #213 Confirmed

Qwen1.5-110B Alibaba (Qwen)

34.2

Rank #234 Confirmed

Summary

  • They share 10 benchmarks with published results for both. Llama 3.1 Nemotron Ultra 253b v1 scores higher in 7 categories and Qwen1.5-110B in 0 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where Llama 3.1 Nemotron Ultra 253b v1 leads 52.2 to 38.0.

Side by side

Llama 3.1 Nemotron Ultra 253b v1Qwen1.5-110B
ProviderNVIDIAAlibaba (Qwen)
Noometry Index36.734.2
Released—2024-04-25
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked1120

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Llama 3.1 Nemotron Ultra 253b v1 leads

Llama 3.1 Nemotron Ultra 253b v1: 38.4 (#177), Qwen1.5-110B: 33.0 (#264)

BenchmarkLlama 3.1 Nemotron Ultra 253b v1Qwen1.5-110B
LMArena Coding13121184
BigCodeBench Instruct—35%
BigCodeBench Complete—44.4%

Agentic & Tool Use Not comparable

Llama 3.1 Nemotron Ultra 253b v1: 15.7 (#149), Qwen1.5-110B: —

BenchmarkLlama 3.1 Nemotron Ultra 253b v1Qwen1.5-110B
Berkeley Function Calling Leaderboard10%—

Reasoning Llama 3.1 Nemotron Ultra 253b v1 leads

Llama 3.1 Nemotron Ultra 253b v1: 26.3 (#134), Qwen1.5-110B: 22.7 (#189)

BenchmarkLlama 3.1 Nemotron Ultra 253b v1Qwen1.5-110B
LMArena Hard Prompts13161168
ForecastBench—57.7

Math Llama 3.1 Nemotron Ultra 253b v1 leads

Llama 3.1 Nemotron Ultra 253b v1: 37.5 (#152), Qwen1.5-110B: 33.7 (#201)

BenchmarkLlama 3.1 Nemotron Ultra 253b v1Qwen1.5-110B
LMArena Math13601185

Knowledge Not comparable

Llama 3.1 Nemotron Ultra 253b v1: —, Qwen1.5-110B: 31.2 (#219)

BenchmarkLlama 3.1 Nemotron Ultra 253b v1Qwen1.5-110B
LMArena Expert—1144

Multilingual Llama 3.1 Nemotron Ultra 253b v1 leads

Llama 3.1 Nemotron Ultra 253b v1: 43.1 (#187), Qwen1.5-110B: 33.6 (#250)

BenchmarkLlama 3.1 Nemotron Ultra 253b v1Qwen1.5-110B
LMArena Non-English12821142
LMArena Russian12841118
LMArena Chinese—1206
LMArena French—1151
LMArena German—1123
LMArena Japanese—1074
LMArena Korean—1044
LMArena Spanish—1142

Instruction Following Llama 3.1 Nemotron Ultra 253b v1 leads

Llama 3.1 Nemotron Ultra 253b v1: 69.0 (#178), Qwen1.5-110B: 60.3 (#252)

BenchmarkLlama 3.1 Nemotron Ultra 253b v1Qwen1.5-110B
LMArena Instruction Following13081158

Long Context Llama 3.1 Nemotron Ultra 253b v1 leads

Llama 3.1 Nemotron Ultra 253b v1: 39.5 (#177), Qwen1.5-110B: 35.1 (#242)

BenchmarkLlama 3.1 Nemotron Ultra 253b v1Qwen1.5-110B
LMArena Longer Query12991157

Writing & Preference Llama 3.1 Nemotron Ultra 253b v1 leads

Llama 3.1 Nemotron Ultra 253b v1: 52.2 (#175), Qwen1.5-110B: 38.0 (#255)

BenchmarkLlama 3.1 Nemotron Ultra 253b v1Qwen1.5-110B
LMArena Text13201175
LMArena Creative Writing13141148
LMArena Multi-Turn13171160

Frequently asked questions

Is Llama 3.1 Nemotron Ultra 253b v1 better than Qwen1.5-110B?

Llama 3.1 Nemotron Ultra 253b v1 is the stronger model overall, scoring 36.7 to 34.2 on the Noometry Index.

Is Llama 3.1 Nemotron Ultra 253b v1 or Qwen1.5-110B better for coding?

Llama 3.1 Nemotron Ultra 253b v1 scores higher on coding benchmarks: 38.4 versus 33.0 in the Noometry coding category.

How many benchmarks do Llama 3.1 Nemotron Ultra 253b v1 and Qwen1.5-110B share?

10 benchmarks have published results for both models. Llama 3.1 Nemotron Ultra 253b v1 has 11 scored results on Noometry and Qwen1.5-110B has 20.

Related comparisons

Go deeper