Noometry

Model comparison

Llama 3.1-8B vs Longcat Flash Chat

Longcat Flash Chat is the stronger model overall, scoring 42.1 to 23.0 on the Noometry Index.

Last verified . 17 shared benchmarks.

Llama 3.1-8B Meta

23.0

Rank #352 Confirmed

Longcat Flash Chat Meituan

42.1

Rank #120 Confirmed

Summary

  • They share 17 benchmarks with published results for both. Llama 3.1-8B scores higher in 0 categories and Longcat Flash Chat in 8 categories; 8 gaps are clear of the uncertainty.
  • The widest gap is in knowledge, where Longcat Flash Chat leads 40.6 to 8.0.

Side by side

Llama 3.1-8BLongcat Flash Chat
ProviderMetaMeituan
Noometry Index23.042.1
Released2024-07-23—
WeightsOpenOpen
Context window128K—
Max output4K—
Input $ / M tokens$0.05—
Output $ / M tokens$0.08—
Results tracked4319

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Longcat Flash Chat leads

Llama 3.1-8B: 20.2 (#340), Longcat Flash Chat: 43.5 (#87)

BenchmarkLlama 3.1-8BLongcat Flash Chat
LMArena Coding11951471
SciCode13.2%—
WeirdML1.7%—
BigCodeBench Instruct32.8%—
BigCodeBench Complete40.5%—
HumanEval+62.8%—
MBPP+55.6%—

Agentic & Tool Use Not comparable

Llama 3.1-8B: 22.5 (#131), Longcat Flash Chat: —

BenchmarkLlama 3.1-8BLongcat Flash Chat
Berkeley Function Calling Leaderboard25.8%—
BALROG15.1%—

Reasoning Longcat Flash Chat leads

Llama 3.1-8B: 14.9 (#321), Longcat Flash Chat: 19.0 (#272)

BenchmarkLlama 3.1-8BLongcat Flash Chat
LMArena Hard Prompts11751440
Kagi LLM Benchmark—43.9%
NYT Connections (extended)—17.7%
CritPt0%—
Chess Puzzles0%—
DTBench50.9%—
LMCA5.4%—
Epoch Capabilities Index116.57—
PIQA81.2%—

Math Longcat Flash Chat leads

Llama 3.1-8B: 10.2 (#317), Longcat Flash Chat: 39.4 (#107)

BenchmarkLlama 3.1-8BLongcat Flash Chat
LMArena Math11791442
OTIS Mock AIME 2024-20251.7%—
Omni-MATH13.7%—
MATH Level 522.9%—
GSM8K82.4%—

Knowledge Longcat Flash Chat leads

Llama 3.1-8B: 8.0 (#307), Longcat Flash Chat: 40.6 (#116)

BenchmarkLlama 3.1-8BLongcat Flash Chat
LMArena Expert11441454
GPQA Diamond27%—
MMLU-Pro40.6%—
GPQA (HELM)24.7%—
BoolQ82.8%—
MMLU56.1%—

Multilingual Longcat Flash Chat leads

Llama 3.1-8B: 34.0 (#249), Longcat Flash Chat: 51.9 (#101)

BenchmarkLlama 3.1-8BLongcat Flash Chat
LMArena Non-English11481404
LMArena Chinese11511465
LMArena French11771456
LMArena German11441408
LMArena Japanese10611373
LMArena Korean10531371
LMArena Russian11581395
LMArena Spanish11691445

Instruction Following Longcat Flash Chat leads

Llama 3.1-8B: 58.9 (#258), Longcat Flash Chat: 74.4 (#96)

BenchmarkLlama 3.1-8BLongcat Flash Chat
LMArena Instruction Following11591411
IFEval74.3%—

Long Context Longcat Flash Chat leads

Llama 3.1-8B: 35.8 (#238), Longcat Flash Chat: 43.5 (#93)

BenchmarkLlama 3.1-8BLongcat Flash Chat
LMArena Longer Query11821425

Writing & Preference Longcat Flash Chat leads

Llama 3.1-8B: 29.7 (#290), Longcat Flash Chat: 61.0 (#91)

BenchmarkLlama 3.1-8BLongcat Flash Chat
LMArena Text11871427
LMArena Creative Writing11541388
LMArena Multi-Turn11721418
EQ-Bench Creative Writing713—
WildBench68.7%—

Frequently asked questions

Is Llama 3.1-8B better than Longcat Flash Chat?

Longcat Flash Chat is the stronger model overall, scoring 42.1 to 23.0 on the Noometry Index.

Is Llama 3.1-8B or Longcat Flash Chat better for coding?

Longcat Flash Chat scores higher on coding benchmarks: 43.5 versus 20.2 in the Noometry coding category.

How many benchmarks do Llama 3.1-8B and Longcat Flash Chat share?

17 benchmarks have published results for both models. Llama 3.1-8B has 43 scored results on Noometry and Longcat Flash Chat has 19.

Related comparisons

Go deeper