Noometry

Model comparison

Llama 3.1-405B vs Yi-1.5-34B

Llama 3.1-405B and Yi-1.5-34B score almost the same on the Noometry Index (30.7 vs 30.6), so choose on price, context window or the category you care about most.

Last verified . 19 shared benchmarks.

Llama 3.1-405B Meta

30.7

Rank #288 Confirmed

Yi-1.5-34B 01.AI

30.6

Rank #289 Confirmed

Summary

  • They share 19 benchmarks with published results for both. Llama 3.1-405B scores higher in 6 categories and Yi-1.5-34B in 2 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Llama 3.1-405B leads 30.4 to 14.8.
  • The biggest single-benchmark swing is MATH Level 5: 49.8% for Llama 3.1-405B and 25.5% for Yi-1.5-34B.

Side by side

Llama 3.1-405BYi-1.5-34B
ProviderMeta01.AI
Noometry Index30.730.6
Released2024-07-232024-05-13
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked4221

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Too close to call

Llama 3.1-405B: 33.1 (#262), Yi-1.5-34B: 32.4 (#272)

BenchmarkLlama 3.1-405BYi-1.5-34B
LMArena Coding12911169
WeirdML21.4%—
BigCodeBench Instruct—33.9%
BigCodeBench Complete—43.8%

Agentic & Tool Use Not comparable

Llama 3.1-405B: 21.0 (#140), Yi-1.5-34B: —

BenchmarkLlama 3.1-405BYi-1.5-34B
TheAgentCompany7.4%—
Cybench7.5%—

Reasoning Yi-1.5-34B leads

Llama 3.1-405B: 16.8 (#300), Yi-1.5-34B: 22.5 (#191)

BenchmarkLlama 3.1-405BYi-1.5-34B
LMArena Hard Prompts12691160
SimpleBench23%—
Kagi LLM Benchmark45%—
DTBench61.4%—
BIG-Bench Hard82.9%—
Epoch Capabilities Index128.75—
ForecastBench59.9—
HellaSwag89.2%—
PIQA85.9%—
WinoGrande89.2%—

Math Yi-1.5-34B leads

Llama 3.1-405B: 18.4 (#290), Yi-1.5-34B: 27.5 (#249)

BenchmarkLlama 3.1-405BYi-1.5-34B
LMArena Math12811182
MATH Level 549.8%25.5%
OTIS Mock AIME 2024-20259.7%—
Omni-MATH24.9%—

Knowledge Llama 3.1-405B leads

Llama 3.1-405B: 30.4 (#227), Yi-1.5-34B: 14.8 (#295)

BenchmarkLlama 3.1-405BYi-1.5-34B
GPQA Diamond50.9%32%
LMArena Expert12431144
MMLU-Pro72.3%—
Confabulations17.6%—
GPQA (HELM)52.2%—
ARC (AI2) Challenge95.3%—
MMLU84.5%—
TriviaQA82.7%—

Multilingual Llama 3.1-405B leads

Llama 3.1-405B: 40.7 (#214), Yi-1.5-34B: 32.3 (#256)

BenchmarkLlama 3.1-405BYi-1.5-34B
LMArena Non-English12481121
LMArena Chinese12421213
LMArena French12791156
LMArena German12521111
LMArena Japanese12081021
LMArena Korean11841005
LMArena Russian12651091
LMArena Spanish12601121

Instruction Following Llama 3.1-405B leads

Llama 3.1-405B: 65.9 (#214), Yi-1.5-34B: 59.2 (#257)

BenchmarkLlama 3.1-405BYi-1.5-34B
LMArena Instruction Following12591139
IFEval81.1%—

Long Context Llama 3.1-405B leads

Llama 3.1-405B: 38.4 (#197), Yi-1.5-34B: 34.6 (#248)

BenchmarkLlama 3.1-405BYi-1.5-34B
LMArena Longer Query12661143

Writing & Preference Llama 3.1-405B leads

Llama 3.1-405B: 38.9 (#251), Yi-1.5-34B: 37.4 (#257)

BenchmarkLlama 3.1-405BYi-1.5-34B
LMArena Text12841173
LMArena Creative Writing12621135
LMArena Multi-Turn12971153
EQ-Bench Creative Writing870—
WildBench78.3%—

Frequently asked questions

Is Llama 3.1-405B better than Yi-1.5-34B?

Llama 3.1-405B and Yi-1.5-34B score almost the same on the Noometry Index (30.7 vs 30.6), so choose on price, context window or the category you care about most.

Is Llama 3.1-405B or Yi-1.5-34B better for coding?

They score almost the same on coding (33.1 vs 32.4); test both on your own repository before choosing.

How many benchmarks do Llama 3.1-405B and Yi-1.5-34B share?

19 benchmarks have published results for both models. Llama 3.1-405B has 42 scored results on Noometry and Yi-1.5-34B has 21.

Related comparisons

Go deeper