Noometry

Model comparison

Mixtral 8x7B vs Tulu 3 (Tülu 3) 70B

Tulu 3 (Tülu 3) 70B is the stronger model overall, scoring 33.0 to 27.1 on the Noometry Index.

Last verified . 13 shared benchmarks.

Mixtral 8x7B Mistral AI

27.1

Rank #334 Confirmed

Tulu 3 (Tülu 3) 70B Allen Institute for AI (Ai2)

33.0

Rank #251 Confirmed

Summary

  • They share 13 benchmarks with published results for both. Mixtral 8x7B scores higher in 1 category and Tulu 3 (Tülu 3) 70B in 7 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Tulu 3 (Tülu 3) 70B leads 25.0 to 11.0.
  • The biggest single-benchmark swing is MATH Level 5: 10% for Mixtral 8x7B and 42.7% for Tulu 3 (Tülu 3) 70B.

Side by side

Mixtral 8x7BTulu 3 (Tülu 3) 70B
ProviderMistral AIAllen Institute for AI (Ai2)
Noometry Index27.133.0
Released2023-12-112024-11-21
WeightsOpenOpen
Context window32K—
Max output32K—
Input $ / M tokens$0.70—
Output $ / M tokens$0.70—
Results tracked3814

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Tulu 3 (Tülu 3) 70B leads

Mixtral 8x7B: 32.8 (#269), Tulu 3 (Tülu 3) 70B: 36.0 (#214)

BenchmarkMixtral 8x7BTulu 3 (Tülu 3) 70B
LMArena Coding11261235
HumanEval+39.6%—
MBPP+49.7%—

Reasoning Tulu 3 (Tülu 3) 70B leads

Mixtral 8x7B: 18.2 (#285), Tulu 3 (Tülu 3) 70B: 23.9 (#169)

BenchmarkMixtral 8x7BTulu 3 (Tülu 3) 70B
LMArena Hard Prompts11151220
DTBench49.6%—
Adversarial NLI55.2%—
Epoch Capabilities Index118.47—
ForecastBench56.3—
HellaSwag86.7%—
PIQA83.6%—
WinoGrande77.2%—

Math Mixtral 8x7B leads

Mixtral 8x7B: 18.8 (#289), Tulu 3 (Tülu 3) 70B: 14.2 (#303)

BenchmarkMixtral 8x7BTulu 3 (Tülu 3) 70B
LMArena Math11471242
MATH Level 510%42.7%
OTIS Mock AIME 2024-2025—4.4%
Omni-MATH10.5%—
GSM8K74.4%—

Knowledge Tulu 3 (Tülu 3) 70B leads

Mixtral 8x7B: 11.0 (#301), Tulu 3 (Tülu 3) 70B: 25.0 (#264)

BenchmarkMixtral 8x7BTulu 3 (Tülu 3) 70B
GPQA Diamond30.6%46.3%
MMLU-Pro33.5%—
GPQA (HELM)29.6%—
LMArena Expert1088—
ARC (AI2) Challenge87.3%—
MMLU70.6%—
OpenBookQA85.8%—
TriviaQA82.2%—

Multilingual Tulu 3 (Tülu 3) 70B leads

Mixtral 8x7B: 29.6 (#266), Tulu 3 (Tülu 3) 70B: 39.9 (#222)

BenchmarkMixtral 8x7BTulu 3 (Tülu 3) 70B
LMArena Non-English10771236
LMArena Chinese10551249
LMArena Russian10901246
LMArena French1166—
LMArena German1114—
LMArena Japanese931—
LMArena Korean968—
LMArena Spanish1111—

Instruction Following Tulu 3 (Tülu 3) 70B leads

Mixtral 8x7B: 51.0 (#297), Tulu 3 (Tülu 3) 70B: 64.8 (#227)

BenchmarkMixtral 8x7BTulu 3 (Tülu 3) 70B
LMArena Instruction Following11091233
IFEval57.5%—

Long Context Tulu 3 (Tülu 3) 70B leads

Mixtral 8x7B: 33.4 (#260), Tulu 3 (Tülu 3) 70B: 37.1 (#222)

BenchmarkMixtral 8x7BTulu 3 (Tülu 3) 70B
LMArena Longer Query11031224

Writing & Preference Tulu 3 (Tülu 3) 70B leads

Mixtral 8x7B: 34.2 (#270), Tulu 3 (Tülu 3) 70B: 45.6 (#223)

BenchmarkMixtral 8x7BTulu 3 (Tülu 3) 70B
LMArena Text11321256
LMArena Creative Writing11091231
LMArena Multi-Turn11151252
WildBench67.3%—

Frequently asked questions

Is Mixtral 8x7B better than Tulu 3 (Tülu 3) 70B?

Tulu 3 (Tülu 3) 70B is the stronger model overall, scoring 33.0 to 27.1 on the Noometry Index.

Is Mixtral 8x7B or Tulu 3 (Tülu 3) 70B better for coding?

Tulu 3 (Tülu 3) 70B scores higher on coding benchmarks: 36.0 versus 32.8 in the Noometry coding category.

How many benchmarks do Mixtral 8x7B and Tulu 3 (Tülu 3) 70B share?

13 benchmarks have published results for both models. Mixtral 8x7B has 38 scored results on Noometry and Tulu 3 (Tülu 3) 70B has 14.

Related comparisons

Go deeper