Noometry

Model comparison

GPT-4 Turbo vs Mixtral 8x7B

GPT-4 Turbo is the stronger model overall, scoring 30.3 to 26.8 on the Noometry Index. Mixtral 8x7B costs 21× less per token, which makes it the better buy when GPT-4 Turbo's lead doesn't matter for your workload.

Last verified . 25 shared benchmarks.

GPT-4 Turbo OpenAI

30.3

Rank #281 Confirmed

Mixtral 8x7B Mistral AI

26.8

Rank #328 Confirmed

Summary

  • They share 25 benchmarks with published results for both. GPT-4 Turbo scores higher in 6 categories and Mixtral 8x7B in 2 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in instruction following, where GPT-4 Turbo leads 66.0 to 51.2.
  • The biggest single-benchmark swing is MATH Level 5: 46.7% for GPT-4 Turbo and 10% for Mixtral 8x7B.
  • Mixtral 8x7B is cheaper at $0.70 / $0.70 per million input/output tokens, against $10 / $30 for GPT-4 Turbo.
  • GPT-4 Turbo accepts more context: 128K tokens versus 32K.
  • Mixtral 8x7B has downloadable open weights; the other is API-only.

Side by side

GPT-4 TurboMixtral 8x7B
ProviderOpenAIMistral AI
Noometry Index30.326.8
Released2023-11-062023-12-11
WeightsProprietaryOpen
Context window128K32K
Max output4K32K
Input $ / M tokens$10$0.70
Output $ / M tokens$30$0.70
Results tracked3638

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding GPT-4 Turbo leads

GPT-4 Turbo: 33.4 (#247), Mixtral 8x7B: 32.2 (#265)

BenchmarkGPT-4 TurboMixtral 8x7B
LMArena Coding12681127
HumanEval+86.6%39.6%
MBPP+73.3%49.7%
WeirdML18%—
BigCodeBench Instruct48.2%—
BigCodeBench Complete58.2%—

Agentic & Tool Use Not comparable

GPT-4 Turbo: —, Mixtral 8x7B: —

BenchmarkGPT-4 TurboMixtral 8x7B
METR Time Horizons36.7%—

Reasoning Mixtral 8x7B leads

GPT-4 Turbo: 14.9 (#311), Mixtral 8x7B: 17.4 (#283)

BenchmarkGPT-4 TurboMixtral 8x7B
LMArena Hard Prompts12511115
DTBench61.6%49.6%
Epoch Capabilities Index127.25118.38
ForecastBench59.456.3
SimpleBench25.1%—
Chess Puzzles6%—
LMCA9.8%—
Adversarial NLI—55.2%
HellaSwag—86.7%
PIQA—83.6%
WinoGrande—77.2%

Math Mixtral 8x7B leads

GPT-4 Turbo: 8.6 (#316), Mixtral 8x7B: 17.2 (#284)

BenchmarkGPT-4 TurboMixtral 8x7B
LMArena Math12721147
MATH Level 546.7%10%
FrontierMath (Tiers 1-3)0.7%—
OTIS Mock AIME 2024-20256.7%—
Omni-MATH—10.5%
GSM8K—74.4%

Knowledge GPT-4 Turbo leads

GPT-4 Turbo: 24.3 (#263), Mixtral 8x7B: 11.0 (#296)

BenchmarkGPT-4 TurboMixtral 8x7B
GPQA Diamond46.6%30.6%
LMArena Expert12231088
MMLU81.3%70.6%
MMLU-Pro—33.5%
Confabulations28.4%—
GPQA (HELM)—29.6%
ARC (AI2) Challenge—87.3%
OpenBookQA—85.8%
TriviaQA—82.2%

Multimodal Not comparable

GPT-4 Turbo: 30.0 (#104), Mixtral 8x7B: —

BenchmarkGPT-4 TurboMixtral 8x7B
LMArena Vision1090—

Multilingual GPT-4 Turbo leads

GPT-4 Turbo: 40.7 (#211), Mixtral 8x7B: 29.8 (#261)

BenchmarkGPT-4 TurboMixtral 8x7B
LMArena Non-English12451077
LMArena Chinese12421055
LMArena French12751165
LMArena German12591114
LMArena Japanese1194931
LMArena Korean1187968
LMArena Russian12591090
LMArena Spanish12601110

Instruction Following GPT-4 Turbo leads

GPT-4 Turbo: 66.0 (#211), Mixtral 8x7B: 51.2 (#292)

BenchmarkGPT-4 TurboMixtral 8x7B
LMArena Instruction Following12501110
IFEval—57.5%

Long Context GPT-4 Turbo leads

GPT-4 Turbo: 38.0 (#200), Mixtral 8x7B: 33.4 (#255)

BenchmarkGPT-4 TurboMixtral 8x7B
LMArena Longer Query12541103

Writing & Preference GPT-4 Turbo leads

GPT-4 Turbo: 47.9 (#201), Mixtral 8x7B: 34.4 (#265)

BenchmarkGPT-4 TurboMixtral 8x7B
LMArena Text12721132
LMArena Creative Writing12691109
LMArena Multi-Turn12671115
WildBench—67.3%

Frequently asked questions

Is GPT-4 Turbo better than Mixtral 8x7B?

GPT-4 Turbo is the stronger model overall, scoring 30.3 to 26.8 on the Noometry Index. Mixtral 8x7B costs 21× less per token, which makes it the better buy when GPT-4 Turbo's lead doesn't matter for your workload.

Which is cheaper, GPT-4 Turbo or Mixtral 8x7B?

Mixtral 8x7B is cheaper. It lists at $0.70 per million input tokens and $0.70 per million output tokens; GPT-4 Turbo lists at $10 and $30.

Is GPT-4 Turbo or Mixtral 8x7B better for coding?

GPT-4 Turbo scores higher on coding benchmarks: 33.4 versus 32.2 in the Noometry coding category.

Which has the bigger context window?

GPT-4 Turbo does, with 128K tokens against 32K.

How many benchmarks do GPT-4 Turbo and Mixtral 8x7B share?

25 benchmarks have published results for both models. GPT-4 Turbo has 36 scored results on Noometry and Mixtral 8x7B has 38.

Related comparisons

Go deeper