Noometry

Model comparison

GPT-4 Turbo vs Mixtral 8x22B

GPT-4 Turbo is the stronger model overall, scoring 30.5 to 27.1 on the Noometry Index. Mixtral 8x22B costs 5.0× less per token, which makes it the better buy when GPT-4 Turbo's lead doesn't matter for your workload.

Last verified . 28 shared benchmarks.

GPT-4 Turbo OpenAI

30.5

Rank #292 Confirmed

Mixtral 8x22B Mistral AI

27.1

Rank #333 Confirmed

Summary

  • They share 28 benchmarks with published results for both. GPT-4 Turbo scores higher in 6 categories and Mixtral 8x22B in 2 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in math, where Mixtral 8x22B leads 22.9 to 9.0.
  • The biggest single-benchmark swing is MATH Level 5: 46.7% for GPT-4 Turbo and 24.2% for Mixtral 8x22B.
  • Mixtral 8x22B is cheaper at $2 / $6 per million input/output tokens, against $10 / $30 for GPT-4 Turbo.
  • GPT-4 Turbo accepts more context: 128K tokens versus 64K.
  • Mixtral 8x22B has downloadable open weights; the other is API-only.

Side by side

GPT-4 TurboMixtral 8x22B
ProviderOpenAIMistral AI
Noometry Index30.527.1
Released2023-11-062024-04-17
WeightsProprietaryOpen
Context window128K64K
Max output4K64K
Input $ / M tokens$10$2
Output $ / M tokens$30$6
Results tracked3634

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding GPT-4 Turbo leads

GPT-4 Turbo: 33.8 (#249), Mixtral 8x22B: 24.2 (#329)

BenchmarkGPT-4 TurboMixtral 8x22B
WeirdML18%3.2%
BigCodeBench Instruct48.2%40.6%
LMArena Coding12681166
BigCodeBench Complete58.2%50.2%
HumanEval+86.6%72%
MBPP+73.3%64.3%

Agentic & Tool Use Not comparable

GPT-4 Turbo: —, Mixtral 8x22B: 23.1 (#127)

BenchmarkGPT-4 TurboMixtral 8x22B
Cybench—7.5%
METR Time Horizons36.7%—

Reasoning Mixtral 8x22B leads

GPT-4 Turbo: 15.3 (#317), Mixtral 8x22B: 19.9 (#248)

BenchmarkGPT-4 TurboMixtral 8x22B
LMArena Hard Prompts12511150
DTBench61.6%55.1%
Epoch Capabilities Index127.25122.03
ForecastBench59.456.3
SimpleBench25.1%—
Chess Puzzles6%—
LMCA9.8%—

Math Mixtral 8x22B leads

GPT-4 Turbo: 9.0 (#322), Mixtral 8x22B: 22.9 (#275)

BenchmarkGPT-4 TurboMixtral 8x22B
LMArena Math12721184
MATH Level 546.7%24.2%
FrontierMath (Tiers 1-3)0.7%—
OTIS Mock AIME 2024-20256.7%—
Omni-MATH—16.3%

Knowledge GPT-4 Turbo leads

GPT-4 Turbo: 24.3 (#268), Mixtral 8x22B: 15.1 (#293)

BenchmarkGPT-4 TurboMixtral 8x22B
GPQA Diamond46.6%34.1%
LMArena Expert12231113
MMLU81.3%77.8%
MMLU-Pro—46%
Confabulations28.4%—
GPQA (HELM)—33.4%

Multimodal Not comparable

GPT-4 Turbo: 30.6 (#110), Mixtral 8x22B: —

BenchmarkGPT-4 TurboMixtral 8x22B
LMArena Vision1090—

Multilingual GPT-4 Turbo leads

GPT-4 Turbo: 40.5 (#216), Mixtral 8x22B: 32.8 (#255)

BenchmarkGPT-4 TurboMixtral 8x22B
LMArena Non-English12451128
LMArena Chinese12421116
LMArena French12761166
LMArena German12591141
LMArena Japanese11941037
LMArena Korean11871057
LMArena Russian12591158
LMArena Spanish12601151

Instruction Following GPT-4 Turbo leads

GPT-4 Turbo: 65.8 (#216), Mixtral 8x22B: 57.7 (#266)

BenchmarkGPT-4 TurboMixtral 8x22B
LMArena Instruction Following12491147
IFEval—72.4%

Long Context GPT-4 Turbo leads

GPT-4 Turbo: 38.0 (#206), Mixtral 8x22B: 34.7 (#247)

BenchmarkGPT-4 TurboMixtral 8x22B
LMArena Longer Query12541144

Writing & Preference GPT-4 Turbo leads

GPT-4 Turbo: 47.7 (#206), Mixtral 8x22B: 36.9 (#262)

BenchmarkGPT-4 TurboMixtral 8x22B
LMArena Text12721162
LMArena Creative Writing12691141
LMArena Multi-Turn12671130
WildBench—71.1%

Frequently asked questions

Is GPT-4 Turbo better than Mixtral 8x22B?

GPT-4 Turbo is the stronger model overall, scoring 30.5 to 27.1 on the Noometry Index. Mixtral 8x22B costs 5.0× less per token, which makes it the better buy when GPT-4 Turbo's lead doesn't matter for your workload.

Which is cheaper, GPT-4 Turbo or Mixtral 8x22B?

Mixtral 8x22B is cheaper. It lists at $2 per million input tokens and $6 per million output tokens; GPT-4 Turbo lists at $10 and $30.

Is GPT-4 Turbo or Mixtral 8x22B better for coding?

GPT-4 Turbo scores higher on coding benchmarks: 33.8 versus 24.2 in the Noometry coding category.

Which has the bigger context window?

GPT-4 Turbo does, with 128K tokens against 64K.

How many benchmarks do GPT-4 Turbo and Mixtral 8x22B share?

28 benchmarks have published results for both models. GPT-4 Turbo has 36 scored results on Noometry and Mixtral 8x22B has 34.

Related comparisons

Go deeper