Model comparison
Falcon-180B vs Phi 3 Medium 4k Instruct
Falcon-180B and Phi 3 Medium 4k Instruct score almost the same on the Noometry Index (32.2 vs 33.0), so choose on price, context window or the category you care about most.
Last verified . 6 shared benchmarks.
Technology Innovation Institute
MicrosoftSummary
- They share 6 benchmarks with published results for both. Falcon-180B scores higher in 0 categories and Phi 3 Medium 4k Instruct in 4 categories; 4 gaps are clear of the uncertainty.
- The widest gap is in multilingual, where Phi 3 Medium 4k Instruct leads 30.6 to 25.2.
Side by side
| Falcon-180B | Phi 3 Medium 4k Instruct | |
|---|---|---|
| Provider | Technology Innovation Institute | Microsoft |
| Noometry Index | 32.2 | 33.0 |
| Released | 2023-09-06 | — |
| Weights | Open | Open |
| Context window | — | — |
| Max output | — | — |
| Input $ / M tokens | — | — |
| Output $ / M tokens | — | — |
| Results tracked | 16 | 17 |
Sponsored placements are available on pages like this one. Advertise on Noometry
Category by category
Coding Not comparable
Falcon-180B: —, Phi 3 Medium 4k Instruct: 32.9 (#268)
| Benchmark | Falcon-180B | Phi 3 Medium 4k Instruct |
|---|---|---|
| LMArena Coding | — | 1130 |
Reasoning Phi 3 Medium 4k Instruct leads
Falcon-180B: 19.1 (#269), Phi 3 Medium 4k Instruct: 21.7 (#214)
| Benchmark | Falcon-180B | Phi 3 Medium 4k Instruct |
|---|---|---|
| LMArena Hard Prompts | 1007 | 1127 |
| Epoch Capabilities Index | 112.13 | — |
| HellaSwag | 89% | — |
| LAMBADA | 79.8% | — |
| PIQA | 84.9% | — |
| WinoGrande | 87.1% | — |
Math Not comparable
Falcon-180B: —, Phi 3 Medium 4k Instruct: 33.4 (#202)
| Benchmark | Falcon-180B | Phi 3 Medium 4k Instruct |
|---|---|---|
| LMArena Math | — | 1173 |
| GSM8K | 54.4% | — |
Knowledge Not comparable
Falcon-180B: —, Phi 3 Medium 4k Instruct: 30.2 (#229)
| Benchmark | Falcon-180B | Phi 3 Medium 4k Instruct |
|---|---|---|
| LMArena Expert | — | 1108 |
| ARC (AI2) Challenge | 67.8% | — |
| BoolQ | 89% | — |
| MMLU | 70.6% | — |
| OpenBookQA | 64.2% | — |
Multilingual Phi 3 Medium 4k Instruct leads
Falcon-180B: 25.2 (#286), Phi 3 Medium 4k Instruct: 30.6 (#263)
| Benchmark | Falcon-180B | Phi 3 Medium 4k Instruct |
|---|---|---|
| LMArena Non-English | 1000 | 1094 |
| LMArena Chinese | — | 1108 |
| LMArena French | — | 1110 |
| LMArena German | — | 1101 |
| LMArena Japanese | — | 1042 |
| LMArena Korean | — | 954 |
| LMArena Russian | — | 1145 |
| LMArena Spanish | — | 1095 |
Instruction Following Phi 3 Medium 4k Instruct leads
Falcon-180B: 53.4 (#286), Phi 3 Medium 4k Instruct: 57.6 (#268)
| Benchmark | Falcon-180B | Phi 3 Medium 4k Instruct |
|---|---|---|
| LMArena Instruction Following | 1047 | 1113 |
Long Context Not comparable
Falcon-180B: —, Phi 3 Medium 4k Instruct: 34.0 (#255)
| Benchmark | Falcon-180B | Phi 3 Medium 4k Instruct |
|---|---|---|
| LMArena Longer Query | — | 1120 |
Writing & Preference Phi 3 Medium 4k Instruct leads
Falcon-180B: 29.1 (#295), Phi 3 Medium 4k Instruct: 34.1 (#272)
| Benchmark | Falcon-180B | Phi 3 Medium 4k Instruct |
|---|---|---|
| LMArena Text | 1054 | 1138 |
| LMArena Creative Writing | 1089 | 1107 |
| LMArena Multi-Turn | 1013 | 1088 |
Frequently asked questions
Is Falcon-180B better than Phi 3 Medium 4k Instruct?
Falcon-180B and Phi 3 Medium 4k Instruct score almost the same on the Noometry Index (32.2 vs 33.0), so choose on price, context window or the category you care about most.
How many benchmarks do Falcon-180B and Phi 3 Medium 4k Instruct share?
6 benchmarks have published results for both models. Falcon-180B has 16 scored results on Noometry and Phi 3 Medium 4k Instruct has 17.