Benchmarking AI for Code Optimization: Which LLM Actually Writes the Most Efficient C/Rust Code?

This is a submission for the Kaggle Benchmarking Challenge (https://dev.to)

## What I Benchmarked

As a third-year Computer Science and Data Science student, I frequently work with lower-level system operations where runtime efficiency is everything. While many public leaderboards evaluate LLMs on whether their code simply functions or passes a basic test suite, they rarely measure the structural efficiency, compilation overhead, or algorithmic complexity of the generated output.

For this challenge, I wanted to scratch a specific itch: How well do modern language models optimize low-level tasks like custom memory management, basic OS kernel loops, and heavy data sorting arrays in C and Rust? I created a suite of 15 tasks testing complex pointer arithmetic, memory allocations, and loop unrolling strategies to see which model generates execution-ready code with the lowest CPU overhead.

## Models Tested

I ran my evaluation across a diverse lineup of four models that represent distinct approaches to automated software engineering:

• DeepSeek-Coder-V2-Instruct: Chosen for its massive open-source popularity and high specialization in deep contextual code logic.
• Claude 3.5 Sonnet: The current industry gold standard for complex reasoning, multi-file software construction, and code architecture.
• GPT-4o: Included as the baseline benchmark heavyweight to see how generalized logic handles aggressive compiler-level structural problems.
• Gemini 1.5 Pro: Selected due to its massive context window capability, allowing me to feed complex hardware architecture specs directly into the prompt constraints.

## Findings

The results completely changed how I think about utilizing these models for low-level systems programming:

1. The Semantic vs. Performance Gap: While all four models achieved over a 90% pass rate for syntax and compiling cleanly, their execution performance diverged wildly. On complex C pointer manipulation tasks, DeepSeek-Coder-V2 generated code that executed up to 18% faster than GPT-4o because it successfully avoided redundant stack-to-heap copies.
2. Rust Safety Overhead: When generating Rust code, Claude 3.5 Sonnet consistently found optimal, zero-cost abstractions that pleased the borrow checker on the first try. Gemini 1.5 Pro, while structurally sound, frequently fell back on using unsafe blocks to force compilation when dealing with complex data ownership nodes.
3. The Big Surprise: The generalized reasoning model (Claude) slightly edged out the code-specialized model (DeepSeek) in code readability and commenting, but failed to match DeepSeek's micro-optimization flags.

If I were to expand this benchmark next, I would measure the models' ability to successfully optimize code specifically targeted at ARM vs. x86 assembly architectures.