Noometry

Model comparison

Llama 3.1-70B vs Step 2 16k Exp 202412

Step 2 16k Exp 202412 is the stronger model overall, scoring 39.2 to 29.6 on the Noometry Index.

Last verified . 12 shared benchmarks.

Llama 3.1-70B Meta

29.6

Rank #308 Confirmed

Step 2 16k Exp 202412 StepFun

39.2

Rank #171 Confirmed

Summary

  • They share 12 benchmarks with published results for both. Llama 3.1-70B scores higher in 0 categories and Step 2 16k Exp 202412 in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where Step 2 16k Exp 202412 leads 36.3 to 13.5.
  • Llama 3.1-70B has downloadable open weights; the other is API-only.

Side by side

Llama 3.1-70BStep 2 16k Exp 202412
ProviderMetaStepFun
Noometry Index29.639.2
Released2024-07-23—
WeightsOpenProprietary
Context window128K—
Max output4K—
Input $ / M tokens$0.40—
Output $ / M tokens$0.40—
Results tracked3512

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Step 2 16k Exp 202412 leads

Llama 3.1-70B: 30.3 (#296), Step 2 16k Exp 202412: 38.5 (#174)

BenchmarkLlama 3.1-70BStep 2 16k Exp 202412
LMArena Coding12601316
WeirdML9%—
BigCodeBench Instruct46.1%—
BigCodeBench Complete54.8%—

Agentic & Tool Use Not comparable

Llama 3.1-70B: 25.1 (#112), Step 2 16k Exp 202412: —

BenchmarkLlama 3.1-70BStep 2 16k Exp 202412
TheAgentCompany6.9%—
BALROG27.9%—

Reasoning Step 2 16k Exp 202412 leads

Llama 3.1-70B: 21.6 (#220), Step 2 16k Exp 202412: 25.9 (#142)

BenchmarkLlama 3.1-70BStep 2 16k Exp 202412
LMArena Hard Prompts12411299
DTBench60%—
LMCA14.8%—
Epoch Capabilities Index125.92—

Math Step 2 16k Exp 202412 leads

Llama 3.1-70B: 13.5 (#304), Step 2 16k Exp 202412: 36.3 (#169)

BenchmarkLlama 3.1-70BStep 2 16k Exp 202412
LMArena Math12521304
OTIS Mock AIME 2024-20253.6%—
Omni-MATH21%—
MATH Level 536.7%—

Knowledge Step 2 16k Exp 202412 leads

Llama 3.1-70B: 24.2 (#269), Step 2 16k Exp 202412: 35.2 (#187)

BenchmarkLlama 3.1-70BStep 2 16k Exp 202412
LMArena Expert12091279
GPQA Diamond44.2%—
MMLU-Pro65.3%—
GPQA (HELM)42.6%—
MMLU80.1%—

Multilingual Step 2 16k Exp 202412 leads

Llama 3.1-70B: 38.8 (#225), Step 2 16k Exp 202412: 43.7 (#181)

BenchmarkLlama 3.1-70BStep 2 16k Exp 202412
LMArena Non-English12191290
LMArena Chinese12151331
LMArena Russian12341324
LMArena French1261—
LMArena German1222—
LMArena Japanese1132—
LMArena Korean1140—
LMArena Spanish1253—

Instruction Following Step 2 16k Exp 202412 leads

Llama 3.1-70B: 65.3 (#223), Step 2 16k Exp 202412: 67.9 (#192)

BenchmarkLlama 3.1-70BStep 2 16k Exp 202412
LMArena Instruction Following12311287
IFEval82.1%—

Long Context Step 2 16k Exp 202412 leads

Llama 3.1-70B: 37.6 (#214), Step 2 16k Exp 202412: 39.7 (#170)

BenchmarkLlama 3.1-70BStep 2 16k Exp 202412
LMArena Longer Query12411307

Writing & Preference Step 2 16k Exp 202412 leads

Llama 3.1-70B: 35.4 (#267), Step 2 16k Exp 202412: 52.3 (#173)

BenchmarkLlama 3.1-70BStep 2 16k Exp 202412
LMArena Text12611321
LMArena Creative Writing12321328
LMArena Multi-Turn12561292
EQ-Bench Creative Writing784—
WildBench75.8%—

Frequently asked questions

Is Llama 3.1-70B better than Step 2 16k Exp 202412?

Step 2 16k Exp 202412 is the stronger model overall, scoring 39.2 to 29.6 on the Noometry Index.

Is Llama 3.1-70B or Step 2 16k Exp 202412 better for coding?

Step 2 16k Exp 202412 scores higher on coding benchmarks: 38.5 versus 30.3 in the Noometry coding category.

How many benchmarks do Llama 3.1-70B and Step 2 16k Exp 202412 share?

12 benchmarks have published results for both models. Llama 3.1-70B has 35 scored results on Noometry and Step 2 16k Exp 202412 has 12.

Related comparisons

Go deeper