Noometry

Model comparison

Claude 2 vs Qwen2.5-Coder (1.5B)

Claude 2 has enough public results to be ranked (#346); Qwen2.5-Coder (1.5B) does not yet, so treat this comparison as directional.

Last verified . 2 shared benchmarks.

Claude 2 Anthropic

25.0

Rank #346 Reported

Qwen2.5-Coder (1.5B) Alibaba (Qwen)

—

Unranked

Summary

  • They share 2 benchmarks with published results for both.
  • Qwen2.5-Coder (1.5B) has downloadable open weights; the other is API-only.

Side by side

Claude 2Qwen2.5-Coder (1.5B)
ProviderAnthropicAlibaba (Qwen)
Noometry Index25.0—
Released2023-07-112024-09-18
WeightsProprietaryOpen
Context window——
Max output——
Input $ / M tokens——
Output $ / M tokens——
Results tracked86

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding Not comparable

Claude 2: —, Qwen2.5-Coder (1.5B): —

BenchmarkClaude 2Qwen2.5-Coder (1.5B)
HumanEval+61.6%—

Reasoning Not comparable

Claude 2: 21.7 (#216), Qwen2.5-Coder (1.5B): —

BenchmarkClaude 2Qwen2.5-Coder (1.5B)
Epoch Capabilities Index120.13113.14
DTBench51.9%—
HellaSwag—76.8%
WinoGrande—72.9%

Math Not comparable

Claude 2: 9.3 (#320), Qwen2.5-Coder (1.5B): —

BenchmarkClaude 2Qwen2.5-Coder (1.5B)
OTIS Mock AIME 2024-20252.5%—
MATH Level 511.7%—
GSM8K—86.7%

Knowledge Not comparable

Claude 2: 16.9 (#287), Qwen2.5-Coder (1.5B): —

BenchmarkClaude 2Qwen2.5-Coder (1.5B)
MMLU78.5%68%
GPQA Diamond34.7%—
ARC (AI2) Challenge—60.9%
TriviaQA87.5%—

Frequently asked questions

Is Claude 2 better than Qwen2.5-Coder (1.5B)?

Claude 2 has enough public results to be ranked (#346); Qwen2.5-Coder (1.5B) does not yet, so treat this comparison as directional.

How many benchmarks do Claude 2 and Qwen2.5-Coder (1.5B) share?

2 benchmarks have published results for both models. Claude 2 has 8 scored results on Noometry and Qwen2.5-Coder (1.5B) has 6.

Related comparisons

Go deeper