Noometry

Model comparison

Gemini 1.5 Flash (May 2024) vs Molmo 2 8b

Molmo 2 8b is the stronger model overall, scoring 39.1 to 33.2 on the Noometry Index.

Last verified . 5 shared benchmarks.

Gemini 1.5 Flash (May 2024) Google

33.2

Rank #246 Confirmed

Molmo 2 8b Allen Institute for AI (Ai2)

39.1

Rank #174 Confirmed

Summary

  • They share 5 benchmarks with published results for both. Gemini 1.5 Flash (May 2024) scores higher in 2 categories and Molmo 2 8b in 3 categories; 2 gaps are clear of the uncertainty.
  • The widest gap is in multimodal, where Gemini 1.5 Flash (May 2024) leads 36.0 to 30.2.
  • Molmo 2 8b has downloadable open weights; the other is API-only.

Side by side

Gemini 1.5 Flash (May 2024)Molmo 2 8b
ProviderGoogleAllen Institute for AI (Ai2)
Noometry Index33.239.1
Released2024-05-14—
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked425

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Gemini 1.5 Flash (May 2024): 34.4 (#236), Molmo 2 8b: —

BenchmarkGemini 1.5 Flash (May 2024)Molmo 2 8b
WeirdML24.9%—
BigCodeBench Instruct43.5%—
LMArena Coding1261—
BigCodeBench Complete55.1%—
HumanEval+75.6%—
MBPP+67.5%—

Agentic & Tool Use Not comparable

Gemini 1.5 Flash (May 2024): 26.6 (#102), Molmo 2 8b: —

BenchmarkGemini 1.5 Flash (May 2024)Molmo 2 8b
BALROG14.6%—

Reasoning Molmo 2 8b leads

Gemini 1.5 Flash (May 2024): 21.7 (#215), Molmo 2 8b: 25.6 (#146)

BenchmarkGemini 1.5 Flash (May 2024)Molmo 2 8b
LMArena Hard Prompts12571287
DTBench53.8%—
Epoch Capabilities Index129.36—
ForecastBench53.9—
PIQA87.5%—

Math Not comparable

Gemini 1.5 Flash (May 2024): 22.1 (#281), Molmo 2 8b: —

BenchmarkGemini 1.5 Flash (May 2024)Molmo 2 8b
OTIS Mock AIME 2024-202516.3%—
Omni-MATH30.4%—
LMArena Math1269—
MATH Level 561.9%—
FrontierMath (Feb 2025 set)0%—
GSM8K82.4%—

Knowledge Not comparable

Gemini 1.5 Flash (May 2024): 26.2 (#260), Molmo 2 8b: —

BenchmarkGemini 1.5 Flash (May 2024)Molmo 2 8b
GPQA Diamond47.3%—
MMLU-Pro67.8%—
GPQA (HELM)43.7%—
LMArena Expert1233—
BoolQ85.8%—
MMLU77.9%—

Multimodal Gemini 1.5 Flash (May 2024) leads

Gemini 1.5 Flash (May 2024): 36.0 (#81), Molmo 2 8b: 30.2 (#112)

BenchmarkGemini 1.5 Flash (May 2024)Molmo 2 8b
LMArena Vision11411081
Video-MME70.3%—
GeoBench76%—

Multilingual Too close to call

Gemini 1.5 Flash (May 2024): 42.9 (#189), Molmo 2 8b: 42.7 (#190)

BenchmarkGemini 1.5 Flash (May 2024)Molmo 2 8b
LMArena Non-English12781276
LMArena Chinese1295—
LMArena French1258—
LMArena German1262—
LMArena Japanese1252—
LMArena Korean1221—
LMArena Russian1288—
LMArena Spanish1243—

Instruction Following Too close to call

Gemini 1.5 Flash (May 2024): 66.8 (#205), Molmo 2 8b: 67.0 (#201)

BenchmarkGemini 1.5 Flash (May 2024)Molmo 2 8b
LMArena Instruction Following12581270
IFEval83.1%—

Long Context Not comparable

Gemini 1.5 Flash (May 2024): 39.0 (#187), Molmo 2 8b: —

BenchmarkGemini 1.5 Flash (May 2024)Molmo 2 8b
LMArena Longer Query1284—

Writing & Preference Too close to call

Gemini 1.5 Flash (May 2024): 48.7 (#196), Molmo 2 8b: 49.4 (#191)

BenchmarkGemini 1.5 Flash (May 2024)Molmo 2 8b
LMArena Text12871288
LMArena Creative Writing1285—
WildBench79.2%—
LMArena Multi-Turn1253—

Frequently asked questions

Is Gemini 1.5 Flash (May 2024) better than Molmo 2 8b?

Molmo 2 8b is the stronger model overall, scoring 39.1 to 33.2 on the Noometry Index.

How many benchmarks do Gemini 1.5 Flash (May 2024) and Molmo 2 8b share?

5 benchmarks have published results for both models. Gemini 1.5 Flash (May 2024) has 42 scored results on Noometry and Molmo 2 8b has 5.

Related comparisons

Go deeper