Noometry

Model comparison

Gemini 1.5 Pro (May 2024) vs INTELLECT-1

Gemini 1.5 Pro (May 2024) has enough public results to be ranked (#261); INTELLECT-1 does not yet, so treat this comparison as directional.

Last verified . 3 shared benchmarks.

Gemini 1.5 Pro (May 2024) Google

32.1

Rank #261 Confirmed

INTELLECT-1 Hugging Face

—

Unranked

Summary

  • They share 3 benchmarks with published results for both.

Side by side

Gemini 1.5 Pro (May 2024)INTELLECT-1
ProviderGoogleHugging Face
Noometry Index32.1—
Released2024-02-152024-11-29
WeightsProprietaryProprietary
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked457

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Gemini 1.5 Pro (May 2024): 34.2 (#241), INTELLECT-1: —

BenchmarkGemini 1.5 Pro (May 2024)INTELLECT-1
WeirdML22.2%—
BigCodeBench Instruct43.8%—
LMArena Coding1294—
BigCodeBench Complete57.5%—
CadEval34%—
HumanEval+79.3%—
MBPP+74.6%—

Agentic & Tool Use Not comparable

Gemini 1.5 Pro (May 2024): 17.9 (#145), INTELLECT-1: —

BenchmarkGemini 1.5 Pro (May 2024)INTELLECT-1
TheAgentCompany3.4%—
Cybench7.5%—
BALROG21%—

Reasoning Not comparable

Gemini 1.5 Pro (May 2024): 12.3 (#338), INTELLECT-1: —

BenchmarkGemini 1.5 Pro (May 2024)INTELLECT-1
BIG-Bench Hard89.2%34.9%
Epoch Capabilities Index131.73100.86
ARC-AGI-20.8%—
SimpleBench27.1%—
LMArena Hard Prompts1296—
DTBench59%—
ForecastBench58.4—
HellaSwag—71.4%
WinoGrande—65.8%

Math Not comparable

Gemini 1.5 Pro (May 2024): 25.8 (#266), INTELLECT-1: —

BenchmarkGemini 1.5 Pro (May 2024)INTELLECT-1
OTIS Mock AIME 2024-202523.1%—
Omni-MATH36.4%—
LMArena Math1315—
MATH Level 570.4%—
GSM8K—38.6%

Knowledge Not comparable

Gemini 1.5 Pro (May 2024): 29.4 (#239), INTELLECT-1: —

BenchmarkGemini 1.5 Pro (May 2024)INTELLECT-1
MMLU86.9%49.9%
GPQA Diamond57.2%—
Humanity's Last Exam4.6%—
MMLU-Pro73.7%—
Confabulations13.5%—
GPQA (HELM)53.4%—
LMArena Expert1279—
ARC (AI2) Challenge—54.5%

Multimodal Not comparable

Gemini 1.5 Pro (May 2024): 36.8 (#77), INTELLECT-1: —

BenchmarkGemini 1.5 Pro (May 2024)INTELLECT-1
LMArena Vision1161—
Video-MME75%—

Multilingual Not comparable

Gemini 1.5 Pro (May 2024): 45.3 (#174), INTELLECT-1: —

BenchmarkGemini 1.5 Pro (May 2024)INTELLECT-1
LMArena Non-English1312—
LMArena Chinese1331—
LMArena French1302—
LMArena German1286—
LMArena Japanese1292—
LMArena Korean1298—
LMArena Russian1320—
LMArena Spanish1311—

Instruction Following Not comparable

Gemini 1.5 Pro (May 2024): 68.6 (#185), INTELLECT-1: —

BenchmarkGemini 1.5 Pro (May 2024)INTELLECT-1
IFEval83.7%—
LMArena Instruction Following1297—

Long Context Not comparable

Gemini 1.5 Pro (May 2024): 39.8 (#169), INTELLECT-1: —

BenchmarkGemini 1.5 Pro (May 2024)INTELLECT-1
LMArena Longer Query1308—

Writing & Preference Not comparable

Gemini 1.5 Pro (May 2024): 52.4 (#172), INTELLECT-1: —

BenchmarkGemini 1.5 Pro (May 2024)INTELLECT-1
LMArena Text1319—
LMArena Creative Writing1333—
WildBench81.3%—
LMArena Multi-Turn1296—

Frequently asked questions

Is Gemini 1.5 Pro (May 2024) better than INTELLECT-1?

Gemini 1.5 Pro (May 2024) has enough public results to be ranked (#261); INTELLECT-1 does not yet, so treat this comparison as directional.

How many benchmarks do Gemini 1.5 Pro (May 2024) and INTELLECT-1 share?

3 benchmarks have published results for both models. Gemini 1.5 Pro (May 2024) has 45 scored results on Noometry and INTELLECT-1 has 7.

Related comparisons

Go deeper