Noometry

Model comparison

ERNIE 5.1 vs Gemini 1.5 Flash (May 2024)

ERNIE 5.1 is the stronger model overall, scoring 43.1 to 32.9 on the Noometry Index.

Last verified . 17 shared benchmarks.

ERNIE 5.1 Baidu

43.1

Rank #87 Confirmed

Gemini 1.5 Flash (May 2024) Google

32.9

Rank #240 Confirmed

Summary

  • They share 17 benchmarks with published results for both. ERNIE 5.1 scores higher in 8 categories and Gemini 1.5 Flash (May 2024) in 0 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in math, where ERNIE 5.1 leads 37.5 to 20.8.

Side by side

ERNIE 5.1Gemini 1.5 Flash (May 2024)
ProviderBaiduGoogle
Noometry Index43.132.9
Released—2024-05-14
WeightsProprietaryProprietary
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked1942

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding ERNIE 5.1 leads

ERNIE 5.1: 43.4 (#78), Gemini 1.5 Flash (May 2024): 33.9 (#233)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Coding14881262
WeirdML—24.9%
BigCodeBench Instruct—43.5%
BigCodeBench Complete—55.1%
HumanEval+—75.6%
MBPP+—67.5%

Agentic & Tool Use Not comparable

ERNIE 5.1: —, Gemini 1.5 Flash (May 2024): 26.3 (#97)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
BALROG—14.6%
LMArena Search1227—

Reasoning Too close to call

ERNIE 5.1: 20.9 (#209), Gemini 1.5 Flash (May 2024): 20.8 (#212)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Hard Prompts14811257
NYT Connections (extended)23.4%—
DTBench—53.8%
Epoch Capabilities Index—129.37
ForecastBench—53.9
PIQA—87.5%

Math ERNIE 5.1 leads

ERNIE 5.1: 37.5 (#96), Gemini 1.5 Flash (May 2024): 20.8 (#274)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Math14811269
OTIS Mock AIME 2024-2025—16.3%
Omni-MATH—30.4%
MATH Level 5—61.9%
FrontierMath (Feb 2025 set)—0%
GSM8K—82.4%

Knowledge ERNIE 5.1 leads

ERNIE 5.1: 41.2 (#103), Gemini 1.5 Flash (May 2024): 26.2 (#253)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Expert14911233
GPQA Diamond—47.3%
MMLU-Pro—67.8%
GPQA (HELM)—43.7%
BoolQ—85.8%
MMLU—77.9%

Multimodal Not comparable

ERNIE 5.1: —, Gemini 1.5 Flash (May 2024): 35.3 (#76)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Vision—1141
Video-MME—70.3%
GeoBench—76%

Multilingual ERNIE 5.1 leads

ERNIE 5.1: 55.7 (#28), Gemini 1.5 Flash (May 2024): 43.1 (#184)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Non-English14551278
LMArena Chinese15071295
LMArena French14861257
LMArena German14701262
LMArena Japanese14221252
LMArena Korean14281220
LMArena Russian14591288
LMArena Spanish14721242

Instruction Following ERNIE 5.1 leads

ERNIE 5.1: 76.9 (#35), Gemini 1.5 Flash (May 2024): 67.0 (#199)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Instruction Following14601258
IFEval—83.1%

Long Context ERNIE 5.1 leads

ERNIE 5.1: 44.7 (#56), Gemini 1.5 Flash (May 2024): 39.0 (#182)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Longer Query14621284

Writing & Preference ERNIE 5.1 leads

ERNIE 5.1: 65.3 (#50), Gemini 1.5 Flash (May 2024): 48.9 (#190)

BenchmarkERNIE 5.1Gemini 1.5 Flash (May 2024)
LMArena Text14681287
LMArena Creative Writing14401285
LMArena Multi-Turn14711253
WildBench—79.2%

Frequently asked questions

Is ERNIE 5.1 better than Gemini 1.5 Flash (May 2024)?

ERNIE 5.1 is the stronger model overall, scoring 43.1 to 32.9 on the Noometry Index.

Is ERNIE 5.1 or Gemini 1.5 Flash (May 2024) better for coding?

ERNIE 5.1 scores higher on coding benchmarks: 43.4 versus 33.9 in the Noometry coding category.

How many benchmarks do ERNIE 5.1 and Gemini 1.5 Flash (May 2024) share?

17 benchmarks have published results for both models. ERNIE 5.1 has 19 scored results on Noometry and Gemini 1.5 Flash (May 2024) has 42.

Related comparisons

Go deeper