Model comparison

Falcon-7B vs Mixtral 8x7B

Mixtral 8x7B has enough public results to be ranked (#334); Falcon-7B does not yet, so treat this comparison as directional.

Last verified . 9 shared benchmarks.

Summary

  • They share 9 benchmarks with published results for both.

Side by side

Falcon-7B and Mixtral 8x7B specifications
Falcon-7BMixtral 8x7B
ProviderTechnology Innovation InstituteMistral AI
Noometry Index—27.1
Released2023-04-242023-12-11
WeightsOpenOpen
Context window—32K
Max output—32K
Input $ / M tokens—$0.70
Output $ / M tokens—$0.70
Results tracked1238

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Falcon-7B: —, Mixtral 8x7B: 32.8 (#269)

Coding benchmarks
BenchmarkFalcon-7BMixtral 8x7B
LMArena Coding—1126
HumanEval+—39.6%
MBPP+—49.7%

Reasoning Not comparable

Falcon-7B: —, Mixtral 8x7B: 18.2 (#285)

Reasoning benchmarks
BenchmarkFalcon-7BMixtral 8x7B
Epoch Capabilities Index95.1118.47
HellaSwag78.1%86.7%
PIQA80.3%83.6%
WinoGrande67.2%77.2%
LMArena Hard Prompts—1115
DTBench—49.6%
Adversarial NLI—55.2%
BIG-Bench Hard28.8%—
ForecastBench—56.3
LAMBADA74.9%—

Math Not comparable

Falcon-7B: —, Mixtral 8x7B: 18.8 (#289)

Math benchmarks
BenchmarkFalcon-7BMixtral 8x7B
GSM8K6.8%74.4%
Omni-MATH—10.5%
LMArena Math—1147
MATH Level 5—10%

Knowledge Not comparable

Falcon-7B: —, Mixtral 8x7B: 11.0 (#301)

Knowledge benchmarks
BenchmarkFalcon-7BMixtral 8x7B
ARC (AI2) Challenge47.9%87.3%
MMLU35%70.6%
OpenBookQA51.6%85.8%
TriviaQA64.6%82.2%
GPQA Diamond—30.6%
MMLU-Pro—33.5%
GPQA (HELM)—29.6%
LMArena Expert—1088
BoolQ75.3%—

Multilingual Not comparable

Falcon-7B: —, Mixtral 8x7B: 29.6 (#266)

Multilingual benchmarks
BenchmarkFalcon-7BMixtral 8x7B
LMArena Non-English—1077
LMArena Chinese—1055
LMArena French—1166
LMArena German—1114
LMArena Japanese—931
LMArena Korean—968
LMArena Russian—1090
LMArena Spanish—1111

Instruction Following Not comparable

Falcon-7B: —, Mixtral 8x7B: 51.0 (#297)

Instruction Following benchmarks
BenchmarkFalcon-7BMixtral 8x7B
IFEval—57.5%
LMArena Instruction Following—1109

Long Context Not comparable

Falcon-7B: —, Mixtral 8x7B: 33.4 (#260)

Long Context benchmarks
BenchmarkFalcon-7BMixtral 8x7B
LMArena Longer Query—1103

Writing & Preference Not comparable

Falcon-7B: —, Mixtral 8x7B: 34.2 (#270)

Writing & Preference benchmarks
BenchmarkFalcon-7BMixtral 8x7B
LMArena Text—1132
LMArena Creative Writing—1109
WildBench—67.3%
LMArena Multi-Turn—1115

Frequently asked questions

Is Falcon-7B better than Mixtral 8x7B?

Mixtral 8x7B has enough public results to be ranked (#334); Falcon-7B does not yet, so treat this comparison as directional.

How many benchmarks do Falcon-7B and Mixtral 8x7B share?

9 benchmarks have published results for both models. Falcon-7B has 12 scored results on Noometry and Mixtral 8x7B has 38.

Related comparisons

Go deeper