Noometry

Model comparison

Mixtral 8x22B vs Tulu 3 (Tülu 3) 70B

Tulu 3 (Tülu 3) 70B is the stronger model overall, scoring 33.0 to 27.1 on the Noometry Index.

Last verified . 13 shared benchmarks.

Mixtral 8x22B Mistral AI

27.1

Rank #333 Confirmed

Tulu 3 (Tülu 3) 70B Allen Institute for AI (Ai2)

33.0

Rank #251 Confirmed

Summary

  • They share 13 benchmarks with published results for both. Mixtral 8x22B scores higher in 1 category and Tulu 3 (Tülu 3) 70B in 7 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in coding, where Tulu 3 (Tülu 3) 70B leads 36.0 to 24.2.
  • The biggest single-benchmark swing is MATH Level 5: 24.2% for Mixtral 8x22B and 42.7% for Tulu 3 (Tülu 3) 70B.

Side by side

Mixtral 8x22BTulu 3 (Tülu 3) 70B
ProviderMistral AIAllen Institute for AI (Ai2)
Noometry Index27.133.0
Released2024-04-172024-11-21
WeightsOpenOpen
Context window64K—
Max output64K—
Input $ / M tokens$2—
Output $ / M tokens$6—
Results tracked3414

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Tulu 3 (Tülu 3) 70B leads

Mixtral 8x22B: 24.2 (#329), Tulu 3 (Tülu 3) 70B: 36.0 (#214)

BenchmarkMixtral 8x22BTulu 3 (Tülu 3) 70B
LMArena Coding11661235
WeirdML3.2%—
BigCodeBench Instruct40.6%—
BigCodeBench Complete50.2%—
HumanEval+72%—
MBPP+64.3%—

Agentic & Tool Use Not comparable

Mixtral 8x22B: 23.1 (#127), Tulu 3 (Tülu 3) 70B: —

BenchmarkMixtral 8x22BTulu 3 (Tülu 3) 70B
Cybench7.5%—

Reasoning Tulu 3 (Tülu 3) 70B leads

Mixtral 8x22B: 19.9 (#248), Tulu 3 (Tülu 3) 70B: 23.9 (#169)

BenchmarkMixtral 8x22BTulu 3 (Tülu 3) 70B
LMArena Hard Prompts11501220
DTBench55.1%—
Epoch Capabilities Index122.03—
ForecastBench56.3—

Math Mixtral 8x22B leads

Mixtral 8x22B: 22.9 (#275), Tulu 3 (Tülu 3) 70B: 14.2 (#303)

BenchmarkMixtral 8x22BTulu 3 (Tülu 3) 70B
LMArena Math11841242
MATH Level 524.2%42.7%
OTIS Mock AIME 2024-2025—4.4%
Omni-MATH16.3%—

Knowledge Tulu 3 (Tülu 3) 70B leads

Mixtral 8x22B: 15.1 (#293), Tulu 3 (Tülu 3) 70B: 25.0 (#264)

BenchmarkMixtral 8x22BTulu 3 (Tülu 3) 70B
GPQA Diamond34.1%46.3%
MMLU-Pro46%—
GPQA (HELM)33.4%—
LMArena Expert1113—
MMLU77.8%—

Multilingual Tulu 3 (Tülu 3) 70B leads

Mixtral 8x22B: 32.8 (#255), Tulu 3 (Tülu 3) 70B: 39.9 (#222)

BenchmarkMixtral 8x22BTulu 3 (Tülu 3) 70B
LMArena Non-English11281236
LMArena Chinese11161249
LMArena Russian11581246
LMArena French1166—
LMArena German1141—
LMArena Japanese1037—
LMArena Korean1057—
LMArena Spanish1151—

Instruction Following Tulu 3 (Tülu 3) 70B leads

Mixtral 8x22B: 57.7 (#266), Tulu 3 (Tülu 3) 70B: 64.8 (#227)

BenchmarkMixtral 8x22BTulu 3 (Tülu 3) 70B
LMArena Instruction Following11471233
IFEval72.4%—

Long Context Tulu 3 (Tülu 3) 70B leads

Mixtral 8x22B: 34.7 (#247), Tulu 3 (Tülu 3) 70B: 37.1 (#222)

BenchmarkMixtral 8x22BTulu 3 (Tülu 3) 70B
LMArena Longer Query11441224

Writing & Preference Tulu 3 (Tülu 3) 70B leads

Mixtral 8x22B: 36.9 (#262), Tulu 3 (Tülu 3) 70B: 45.6 (#223)

BenchmarkMixtral 8x22BTulu 3 (Tülu 3) 70B
LMArena Text11621256
LMArena Creative Writing11411231
LMArena Multi-Turn11301252
WildBench71.1%—

Frequently asked questions

Is Mixtral 8x22B better than Tulu 3 (Tülu 3) 70B?

Tulu 3 (Tülu 3) 70B is the stronger model overall, scoring 33.0 to 27.1 on the Noometry Index.

Is Mixtral 8x22B or Tulu 3 (Tülu 3) 70B better for coding?

Tulu 3 (Tülu 3) 70B scores higher on coding benchmarks: 36.0 versus 24.2 in the Noometry coding category.

How many benchmarks do Mixtral 8x22B and Tulu 3 (Tülu 3) 70B share?

13 benchmarks have published results for both models. Mixtral 8x22B has 34 scored results on Noometry and Tulu 3 (Tülu 3) 70B has 14.

Related comparisons

Go deeper