Noometry

Model comparison

Llama 3.1 Tulu 3 8b vs Phi 3 Mini 4k Instruct

Llama 3.1 Tulu 3 8b is the stronger model overall, scoring 35.7 to 27.9 on the Noometry Index.

Last verified . 11 shared benchmarks.

Llama 3.1 Tulu 3 8b Allen Institute for AI (Ai2)

35.7

Rank #224 Confirmed

Phi 3 Mini 4k Instruct Microsoft

27.9

Rank #328 Confirmed

Summary

  • They share 11 benchmarks with published results for both. Llama 3.1 Tulu 3 8b scores higher in 7 categories and Phi 3 Mini 4k Instruct in 0 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in instruction following, where Llama 3.1 Tulu 3 8b leads 61.3 to 47.7.

Side by side

Llama 3.1 Tulu 3 8bPhi 3 Mini 4k Instruct
ProviderAllen Institute for AI (Ai2)Microsoft
Noometry Index35.727.9
Released—2024-04-23
WeightsOpenOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked1135

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Llama 3.1 Tulu 3 8b leads

Llama 3.1 Tulu 3 8b: 34.4 (#235), Phi 3 Mini 4k Instruct: 26.6 (#323)

BenchmarkLlama 3.1 Tulu 3 8bPhi 3 Mini 4k Instruct
LMArena Coding11831093
LiveBench Coding—15.5%
HumanEval+—59.1%
MBPP+—54.2%

Reasoning Llama 3.1 Tulu 3 8b leads

Llama 3.1 Tulu 3 8b: 22.8 (#188), Phi 3 Mini 4k Instruct: 14.1 (#328)

BenchmarkLlama 3.1 Tulu 3 8bPhi 3 Mini 4k Instruct
LMArena Hard Prompts11741072
Chess Puzzles—0%
LiveBench Reasoning—26.8%
LiveBench Data Analysis—34.7%
Adversarial NLI—52.8%
BIG-Bench Hard—71.7%
HellaSwag—76.7%
LiveBench—22.4%
WinoGrande—70.8%

Math Llama 3.1 Tulu 3 8b leads

Llama 3.1 Tulu 3 8b: 33.9 (#198), Phi 3 Mini 4k Instruct: 26.6 (#257)

BenchmarkLlama 3.1 Tulu 3 8bPhi 3 Mini 4k Instruct
LMArena Math11951111
LiveBench Math—15.7%

Knowledge Not comparable

Llama 3.1 Tulu 3 8b: —, Phi 3 Mini 4k Instruct: 28.5 (#246)

BenchmarkLlama 3.1 Tulu 3 8bPhi 3 Mini 4k Instruct
LMArena Expert—1045
ARC (AI2) Challenge—84.9%
MMLU—68.8%
OpenBookQA—88%
TriviaQA—64%

Multilingual Llama 3.1 Tulu 3 8b leads

Llama 3.1 Tulu 3 8b: 35.4 (#246), Phi 3 Mini 4k Instruct: 26.3 (#280)

BenchmarkLlama 3.1 Tulu 3 8bPhi 3 Mini 4k Instruct
LMArena Non-English11691021
LMArena Chinese11761021
LMArena Russian11931022
LMArena French—1076
LMArena German—1044
LMArena Japanese—935
LMArena Korean—905
LMArena Spanish—1085

Instruction Following Llama 3.1 Tulu 3 8b leads

Llama 3.1 Tulu 3 8b: 61.3 (#246), Phi 3 Mini 4k Instruct: 47.7 (#303)

BenchmarkLlama 3.1 Tulu 3 8bPhi 3 Mini 4k Instruct
LMArena Instruction Following11741053
LiveBench Instruction Following—39.1%

Long Context Llama 3.1 Tulu 3 8b leads

Llama 3.1 Tulu 3 8b: 35.8 (#239), Phi 3 Mini 4k Instruct: 31.7 (#276)

BenchmarkLlama 3.1 Tulu 3 8bPhi 3 Mini 4k Instruct
LMArena Longer Query11811044

Writing & Preference Llama 3.1 Tulu 3 8b leads

Llama 3.1 Tulu 3 8b: 39.7 (#245), Phi 3 Mini 4k Instruct: 27.6 (#300)

BenchmarkLlama 3.1 Tulu 3 8bPhi 3 Mini 4k Instruct
LMArena Text11931073
LMArena Creative Writing11821037
LMArena Multi-Turn11541018
LiveBench Language—9.2%

Frequently asked questions

Is Llama 3.1 Tulu 3 8b better than Phi 3 Mini 4k Instruct?

Llama 3.1 Tulu 3 8b is the stronger model overall, scoring 35.7 to 27.9 on the Noometry Index.

Is Llama 3.1 Tulu 3 8b or Phi 3 Mini 4k Instruct better for coding?

Llama 3.1 Tulu 3 8b scores higher on coding benchmarks: 34.4 versus 26.6 in the Noometry coding category.

How many benchmarks do Llama 3.1 Tulu 3 8b and Phi 3 Mini 4k Instruct share?

11 benchmarks have published results for both models. Llama 3.1 Tulu 3 8b has 11 scored results on Noometry and Phi 3 Mini 4k Instruct has 35.

Related comparisons

Go deeper