Model comparison
DeepSeek Coder 1.3B vs Gemma 7B
Gemma 7B has enough public results to be ranked (#299); DeepSeek Coder 1.3B does not yet, so treat this comparison as directional.
Last verified . 7 shared benchmarks.
DeepSeek
GoogleSummary
- They share 7 benchmarks with published results for both. DeepSeek Coder 1.3B scores higher in 1 category and Gemma 7B in 0 categories, but none of those gaps is larger than the uncertainty.
Side by side
| DeepSeek Coder 1.3B | Gemma 7B | |
|---|---|---|
| Provider | DeepSeek | |
| Noometry Index | 35.0 | 30.0 |
| Released | 2023-11-02 | 2024-02-21 |
| Weights | Open | Open |
| Context window | — | — |
| Max output | — | — |
| Input $ / M tokens | — | — |
| Output $ / M tokens | — | — |
| Results tracked | 9 | 27 |
Sponsored placements are available on pages like this one. Advertise on Noometry
Category by category
Coding Too close to call
DeepSeek Coder 1.3B: 31.2 (#287), Gemma 7B: 30.5 (#294)
| Benchmark | DeepSeek Coder 1.3B | Gemma 7B |
|---|---|---|
| HumanEval+ | 60.4% | 28.7% |
| MBPP+ | 54.8% | 43.4% |
| BigCodeBench Instruct | 22.8% | — |
| LMArena Coding | — | 1048 |
| BigCodeBench Complete | 29.6% | — |
Reasoning Not comparable
DeepSeek Coder 1.3B: —, Gemma 7B: 19.9 (#249)
| Benchmark | DeepSeek Coder 1.3B | Gemma 7B |
|---|---|---|
| Epoch Capabilities Index | 63.6 | 111.99 |
| WinoGrande | 53.3% | 79% |
| LMArena Hard Prompts | — | 1042 |
| Adversarial NLI | — | 48.7% |
| BIG-Bench Hard | — | 55.1% |
| HellaSwag | — | 82.2% |
| PIQA | — | 81.2% |
Math Not comparable
DeepSeek Coder 1.3B: —, Gemma 7B: 31.2 (#228)
| Benchmark | DeepSeek Coder 1.3B | Gemma 7B |
|---|---|---|
| GSM8K | 4.4% | 46.4% |
| LMArena Math | — | 1066 |
Knowledge Not comparable
DeepSeek Coder 1.3B: —, Gemma 7B: 27.3 (#252)
| Benchmark | DeepSeek Coder 1.3B | Gemma 7B |
|---|---|---|
| ARC (AI2) Challenge | 25.4% | 78.3% |
| MMLU | 25.8% | 66.1% |
| LMArena Expert | — | 1001 |
| BoolQ | — | 83.2% |
| OpenBookQA | — | 78.6% |
| TriviaQA | — | 72.3% |
Multilingual Not comparable
DeepSeek Coder 1.3B: —, Gemma 7B: 25.1 (#287)
| Benchmark | DeepSeek Coder 1.3B | Gemma 7B |
|---|---|---|
| LMArena Non-English | — | 999 |
| LMArena Chinese | — | 1035 |
| LMArena French | — | 1025 |
| LMArena Russian | — | 993 |
Instruction Following Not comparable
DeepSeek Coder 1.3B: —, Gemma 7B: 51.5 (#295)
| Benchmark | DeepSeek Coder 1.3B | Gemma 7B |
|---|---|---|
| LMArena Instruction Following | — | 1017 |
Long Context Not comparable
DeepSeek Coder 1.3B: —, Gemma 7B: 31.1 (#282)
| Benchmark | DeepSeek Coder 1.3B | Gemma 7B |
|---|---|---|
| LMArena Longer Query | — | 1022 |
Writing & Preference Not comparable
DeepSeek Coder 1.3B: —, Gemma 7B: 27.1 (#302)
| Benchmark | DeepSeek Coder 1.3B | Gemma 7B |
|---|---|---|
| LMArena Text | — | 1056 |
| LMArena Creative Writing | — | 1024 |
| LMArena Multi-Turn | — | 963 |
Frequently asked questions
Is DeepSeek Coder 1.3B better than Gemma 7B?
Gemma 7B has enough public results to be ranked (#299); DeepSeek Coder 1.3B does not yet, so treat this comparison as directional.
Is DeepSeek Coder 1.3B or Gemma 7B better for coding?
They score almost the same on coding (31.2 vs 30.5); test both on your own repository before choosing.
How many benchmarks do DeepSeek Coder 1.3B and Gemma 7B share?
7 benchmarks have published results for both models. DeepSeek Coder 1.3B has 9 scored results on Noometry and Gemma 7B has 27.