Noometry

Model comparison

Llama 3.1-8B vs Tulu 3 (Tülu 3) 70B

Tulu 3 (Tülu 3) 70B is the stronger model overall, scoring 33.0 to 23.0 on the Noometry Index.

Last verified . 14 shared benchmarks.

Llama 3.1-8B Meta

23.0

Rank #352 Confirmed

Tulu 3 (Tülu 3) 70B Allen Institute for AI (Ai2)

33.0

Rank #251 Confirmed

Summary

  • They share 14 benchmarks with published results for both. Llama 3.1-8B scores higher in 0 categories and Tulu 3 (Tülu 3) 70B in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Tulu 3 (Tülu 3) 70B leads 25.0 to 8.0.
  • The biggest single-benchmark swing is MATH Level 5: 22.9% for Llama 3.1-8B and 42.7% for Tulu 3 (Tülu 3) 70B.

Side by side

Llama 3.1-8BTulu 3 (Tülu 3) 70B
ProviderMetaAllen Institute for AI (Ai2)
Noometry Index23.033.0
Released2024-07-232024-11-21
WeightsOpenOpen
Context window128K—
Max output4K—
Input $ / M tokens$0.05—
Output $ / M tokens$0.08—
Results tracked4314

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Tulu 3 (Tülu 3) 70B leads

Llama 3.1-8B: 20.2 (#340), Tulu 3 (Tülu 3) 70B: 36.0 (#214)

BenchmarkLlama 3.1-8BTulu 3 (Tülu 3) 70B
LMArena Coding11951235
SciCode13.2%—
WeirdML1.7%—
BigCodeBench Instruct32.8%—
BigCodeBench Complete40.5%—
HumanEval+62.8%—
MBPP+55.6%—

Agentic & Tool Use Not comparable

Llama 3.1-8B: 22.5 (#131), Tulu 3 (Tülu 3) 70B: —

BenchmarkLlama 3.1-8BTulu 3 (Tülu 3) 70B
Berkeley Function Calling Leaderboard25.8%—
BALROG15.1%—

Reasoning Tulu 3 (Tülu 3) 70B leads

Llama 3.1-8B: 14.9 (#321), Tulu 3 (Tülu 3) 70B: 23.9 (#169)

BenchmarkLlama 3.1-8BTulu 3 (Tülu 3) 70B
LMArena Hard Prompts11751220
CritPt0%—
Chess Puzzles0%—
DTBench50.9%—
LMCA5.4%—
Epoch Capabilities Index116.57—
PIQA81.2%—

Math Tulu 3 (Tülu 3) 70B leads

Llama 3.1-8B: 10.2 (#317), Tulu 3 (Tülu 3) 70B: 14.2 (#303)

BenchmarkLlama 3.1-8BTulu 3 (Tülu 3) 70B
OTIS Mock AIME 2024-20251.7%4.4%
LMArena Math11791242
MATH Level 522.9%42.7%
Omni-MATH13.7%—
GSM8K82.4%—

Knowledge Tulu 3 (Tülu 3) 70B leads

Llama 3.1-8B: 8.0 (#307), Tulu 3 (Tülu 3) 70B: 25.0 (#264)

BenchmarkLlama 3.1-8BTulu 3 (Tülu 3) 70B
GPQA Diamond27%46.3%
MMLU-Pro40.6%—
GPQA (HELM)24.7%—
LMArena Expert1144—
BoolQ82.8%—
MMLU56.1%—

Multilingual Tulu 3 (Tülu 3) 70B leads

Llama 3.1-8B: 34.0 (#249), Tulu 3 (Tülu 3) 70B: 39.9 (#222)

BenchmarkLlama 3.1-8BTulu 3 (Tülu 3) 70B
LMArena Non-English11481236
LMArena Chinese11511249
LMArena Russian11581246
LMArena French1177—
LMArena German1144—
LMArena Japanese1061—
LMArena Korean1053—
LMArena Spanish1169—

Instruction Following Tulu 3 (Tülu 3) 70B leads

Llama 3.1-8B: 58.9 (#258), Tulu 3 (Tülu 3) 70B: 64.8 (#227)

BenchmarkLlama 3.1-8BTulu 3 (Tülu 3) 70B
LMArena Instruction Following11591233
IFEval74.3%—

Long Context Tulu 3 (Tülu 3) 70B leads

Llama 3.1-8B: 35.8 (#238), Tulu 3 (Tülu 3) 70B: 37.1 (#222)

BenchmarkLlama 3.1-8BTulu 3 (Tülu 3) 70B
LMArena Longer Query11821224

Writing & Preference Tulu 3 (Tülu 3) 70B leads

Llama 3.1-8B: 29.7 (#290), Tulu 3 (Tülu 3) 70B: 45.6 (#223)

BenchmarkLlama 3.1-8BTulu 3 (Tülu 3) 70B
LMArena Text11871256
LMArena Creative Writing11541231
LMArena Multi-Turn11721252
EQ-Bench Creative Writing713—
WildBench68.7%—

Frequently asked questions

Is Llama 3.1-8B better than Tulu 3 (Tülu 3) 70B?

Tulu 3 (Tülu 3) 70B is the stronger model overall, scoring 33.0 to 23.0 on the Noometry Index.

Is Llama 3.1-8B or Tulu 3 (Tülu 3) 70B better for coding?

Tulu 3 (Tülu 3) 70B scores higher on coding benchmarks: 36.0 versus 20.2 in the Noometry coding category.

How many benchmarks do Llama 3.1-8B and Tulu 3 (Tülu 3) 70B share?

14 benchmarks have published results for both models. Llama 3.1-8B has 43 scored results on Noometry and Tulu 3 (Tülu 3) 70B has 14.

Related comparisons

Go deeper