Noometry

Model comparison

Codellama 34b Instruct vs o1

o1 is the stronger model overall, scoring 40.6 to 30.2 on the Noometry Index.

Last verified . 12 shared benchmarks.

Codellama 34b Instruct Meta

30.2

Rank #285 Confirmed

o1 OpenAI

40.6

Rank #140 Confirmed

Summary

  • They share 12 benchmarks with published results for both. Codellama 34b Instruct scores higher in 0 categories and o1 in 7 categories; 7 gaps are clear of the uncertainty.
  • The widest gap is in writing & preference, where o1 leads 55.8 to 28.4.
  • Codellama 34b Instruct has downloadable open weights; the other is API-only.

Side by side

Codellama 34b Instructo1
ProviderMetaOpenAI
Noometry Index30.240.6
Released—2024-09-12
WeightsOpenProprietary
Context window—200K
Max output—100K
Input $ / M tokens—$15
Output $ / M tokens—$60
Results tracked1452

Sponsored placements are available on pages like this one. Advertise on Noometry

Category by category

Coding o1 leads

Codellama 34b Instruct: 28.0 (#310), o1: 45.7 (#64)

BenchmarkCodellama 34b Instructo1
LMArena Coding10461367
HumanEval+43.9%89%
MBPP+56.3%80.2%
Aider Polyglot—61.7%
WeirdML—47.6%
BigCodeBench Instruct29%—
LiveBench Coding—69.7%
BigCodeBench Complete37.1%—
CadEval—56%

Agentic & Tool Use Not comparable

Codellama 34b Instruct: —, o1: 24.3 (#111)

BenchmarkCodellama 34b Instructo1
Cybench—10%
METR Time Horizons—51.1%

Reasoning o1 leads

Codellama 34b Instruct: 18.8 (#250), o1: 27.5 (#103)

BenchmarkCodellama 34b Instructo1
LMArena Hard Prompts10321372
SimpleBench—41.7%
ARC-AGI-1—30.7%
Chess Puzzles—15%
EnigmaEval—5.7%
LiveBench Reasoning—91.6%
DTBench—74.7%
LiveBench Data Analysis—65.5%
LMCA—22.3%
Epoch Capabilities Index—141.92
LiveBench—75.7%

Math o1 leads

Codellama 34b Instruct: 28.5 (#227), o1: 34.9 (#150)

BenchmarkCodellama 34b Instructo1
LMArena Math10561388
FrontierMath (Tiers 1-3)—14.7%
OTIS Mock AIME 2024-2025—73.3%
LiveBench Math—80.3%
MATH Level 5—94.7%
FrontierMath (Feb 2025 set)—9.3%

Knowledge Not comparable

Codellama 34b Instruct: —, o1: 41.5 (#101)

BenchmarkCodellama 34b Instructo1
GPQA Diamond—76.8%
Humanity's Last Exam—8%
SimpleQA Verified—41.1%
Confabulations—11.7%
LMArena Expert—1362

Multimodal Not comparable

Codellama 34b Instruct: —, o1: 33.6 (#87)

BenchmarkCodellama 34b Instructo1
LMArena Vision—1168
GeoBench—80%
VPCT—37%
SpatialViz-Bench—41.4%

Multilingual o1 leads

Codellama 34b Instruct: 25.9 (#279), o1: 48.8 (#137)

BenchmarkCodellama 34b Instructo1
LMArena Non-English10111359
LMArena Chinese9761394
LMArena French—1343
LMArena German—1337
LMArena Japanese—1346
LMArena Korean—1396
LMArena Russian—1356
LMArena Spanish—1345

Instruction Following o1 leads

Codellama 34b Instruct: 52.4 (#286), o1: 75.0 (#82)

BenchmarkCodellama 34b Instructo1
LMArena Instruction Following10281368
LiveBench Instruction Following—81.5%

Long Context o1 leads

Codellama 34b Instruct: 30.8 (#279), o1: 50.3 (#9)

BenchmarkCodellama 34b Instructo1
LMArena Longer Query10141378
Fiction.LiveBench—83.3%

Writing & Preference o1 leads

Codellama 34b Instruct: 28.4 (#292), o1: 55.8 (#139)

BenchmarkCodellama 34b Instructo1
LMArena Text10661366
LMArena Creative Writing10321347
LMArena Multi-Turn10151368
Short-Story Creative Writing—70.2%
LiveBench Language—65.4%

Frequently asked questions

Is Codellama 34b Instruct better than o1?

o1 is the stronger model overall, scoring 40.6 to 30.2 on the Noometry Index.

Is Codellama 34b Instruct or o1 better for coding?

o1 scores higher on coding benchmarks: 45.7 versus 28.0 in the Noometry coding category.

How many benchmarks do Codellama 34b Instruct and o1 share?

12 benchmarks have published results for both models. Codellama 34b Instruct has 14 scored results on Noometry and o1 has 52.

Related comparisons

Go deeper