Noometry

Model comparison

Llama 3.1-405B vs Step 3.5 Flash

Step 3.5 Flash is the stronger model overall, scoring 42.3 to 30.7 on the Noometry Index.

Last verified . 17 shared benchmarks.

Llama 3.1-405B Meta

30.7

Rank #288 Confirmed

Step 3.5 Flash StepFun

42.3

Rank #116 Confirmed

Summary

  • They share 17 benchmarks with published results for both. Llama 3.1-405B scores higher in 0 categories and Step 3.5 Flash in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where Step 3.5 Flash leads 42.6 to 18.4.

Side by side

Llama 3.1-405BStep 3.5 Flash
ProviderMetaStepFun
Noometry Index30.742.3
Released2024-07-232026-01-29
WeightsOpenOpen
Context window—256K
Max output—256K
Input $ / M tokens—$0.10
Output $ / M tokens—$0.30
Results tracked4219

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Step 3.5 Flash leads

Llama 3.1-405B: 33.1 (#262), Step 3.5 Flash: 42.4 (#105)

BenchmarkLlama 3.1-405BStep 3.5 Flash
LMArena Coding12911436
WeirdML21.4%—

Agentic & Tool Use Not comparable

Llama 3.1-405B: 21.0 (#140), Step 3.5 Flash: —

BenchmarkLlama 3.1-405BStep 3.5 Flash
TheAgentCompany7.4%—
Cybench7.5%—

Reasoning Step 3.5 Flash leads

Llama 3.1-405B: 16.8 (#300), Step 3.5 Flash: 22.2 (#202)

BenchmarkLlama 3.1-405BStep 3.5 Flash
LMArena Hard Prompts12691411
SimpleBench23%—
Kagi LLM Benchmark45%—
NYT Connections (extended)—28.4%
DTBench61.4%—
BIG-Bench Hard82.9%—
Epoch Capabilities Index128.75—
ForecastBench59.9—
HellaSwag89.2%—
PIQA85.9%—
WinoGrande89.2%—

Math Step 3.5 Flash leads

Llama 3.1-405B: 18.4 (#290), Step 3.5 Flash: 42.6 (#84)

BenchmarkLlama 3.1-405BStep 3.5 Flash
LMArena Math12811408
MathArena Final-Answer Competitions—66.8%
OTIS Mock AIME 2024-20259.7%—
Omni-MATH24.9%—
MATH Level 549.8%—

Knowledge Step 3.5 Flash leads

Llama 3.1-405B: 30.4 (#227), Step 3.5 Flash: 39.6 (#132)

BenchmarkLlama 3.1-405BStep 3.5 Flash
LMArena Expert12431421
GPQA Diamond50.9%—
MMLU-Pro72.3%—
Confabulations17.6%—
GPQA (HELM)52.2%—
ARC (AI2) Challenge95.3%—
MMLU84.5%—
TriviaQA82.7%—

Multilingual Step 3.5 Flash leads

Llama 3.1-405B: 40.7 (#214), Step 3.5 Flash: 50.5 (#119)

BenchmarkLlama 3.1-405BStep 3.5 Flash
LMArena Non-English12481385
LMArena Chinese12421447
LMArena French12791421
LMArena German12521405
LMArena Japanese12081354
LMArena Korean11841352
LMArena Russian12651385
LMArena Spanish12601419

Instruction Following Step 3.5 Flash leads

Llama 3.1-405B: 65.9 (#214), Step 3.5 Flash: 73.1 (#124)

BenchmarkLlama 3.1-405BStep 3.5 Flash
LMArena Instruction Following12591385
IFEval81.1%—

Long Context Step 3.5 Flash leads

Llama 3.1-405B: 38.4 (#197), Step 3.5 Flash: 42.8 (#117)

BenchmarkLlama 3.1-405BStep 3.5 Flash
LMArena Longer Query12661402

Writing & Preference Step 3.5 Flash leads

Llama 3.1-405B: 38.9 (#251), Step 3.5 Flash: 58.8 (#113)

BenchmarkLlama 3.1-405BStep 3.5 Flash
LMArena Text12841403
LMArena Creative Writing12621357
LMArena Multi-Turn12971405
EQ-Bench Creative Writing870—
WildBench78.3%—

Frequently asked questions

Is Llama 3.1-405B better than Step 3.5 Flash?

Step 3.5 Flash is the stronger model overall, scoring 42.3 to 30.7 on the Noometry Index.

Is Llama 3.1-405B or Step 3.5 Flash better for coding?

Step 3.5 Flash scores higher on coding benchmarks: 42.4 versus 33.1 in the Noometry coding category.

How many benchmarks do Llama 3.1-405B and Step 3.5 Flash share?

17 benchmarks have published results for both models. Llama 3.1-405B has 42 scored results on Noometry and Step 3.5 Flash has 19.

Related comparisons

Go deeper