LegacyShop: Can an AI maintain a real codebase? (Kaggle Benchmark)

This is a submission for the Kaggle Benchmarking Challenge.

Team: @g00ds0ul · @joyy-dev
Benchmark: https://www.kaggle.com/benchmarks/iseoluwaolowogoke/legacyshop-can-an-ai-maintain-a-real-codebase

The question

Most coding benchmarks give a model short puzzles that have nothing to do with each other. That isn't real engineering work. Real engineering is being handed a messy codebase that nobody fully owns and being asked to keep it running.

This started with a real itch: I measure RAM usage in my own app, and AI assistants kept giving me confident but wrong answers about memory. Joy kept seeing models guess instead of asking. We wanted to know whether AI can do the boring, real part of software work: looking after old code.

So we asked one focused question:

If you hand an AI a purpose-built, legacy-style codebase with realistic cross-file maintenance problems, can it maintain it?

To us, "maintaining" a codebase means four skills:

1. Fix bugs: find the real cause, not just the symptom
2. Reason about memory: know what the code actually costs at runtime
3. Know when to ask: stop and ask instead of guessing
4. Judge performance: work out real complexity across files

All four tasks use the same repo. A model can't score well by being good at one kind of puzzle. It has to understand this one codebase.

The repo: built to feel legacy

legacy_shop is the order-handling code for a fictional 2017 Lagos online shop. Its README starts with: "Nobody fully owns this any more."

It's small on purpose: 33 Python files, about 360 lines, plus a unittest suite. The full repo fit within the models’ context budgets, reducing the risk of truncation; this does not rule out attention, prompt, or other input-processing failures. It has the usual legacy problems:

# config.py
# Shop settings. Some of these are no longer used.
VAT_RATE = 0.075
BULK_QTY = 10           # 10 or more of one item gets the bulk rate
MAX_COUPON_SHARE = 0.5  # a coupon can never take more than half the subtotal
LEGACY_MODE = False
OLD_SHIPPING_FLAT = 1_500     # unused since 2019
LOYALTY_POINTS_PER_1000 = 1   # 1 point per FULL 1,000 naira of subtotal
REFUND_WINDOW_DAYS = 14       # up to and including day 14 after delivery

• Dead code that looks live: pricing_old.py is marked "DEPRECATED (2019). Not imported anywhere." It still sits next to the real pricing.py.
• Mixed money units: amounts are in kobo in orders.py and reports.py, and in naira everywhere else (1 naira = 100 kobo). One missed conversion is off by 100×.
• Rules hidden in docstrings: the README says "If code and docstring disagree, the docstring is the rule."
• Logic spread across files: one order goes api → cart → pricing/shipping → orders → utils.

The four tasks (69 items)

Scoring: each task score is the share of items passed. The overall score is the average of the four tasks. All answers are checked by code, not by an LLM judge.

Making it realistic and fair

• Real failures, not trick questions. Every trap is something we've seen in real codebases: a deprecated file left behind, mixed units, a rule only written in a comment.
• No memorization. The repo was written from scratch, so no model has seen it.
• Hardened until it stopped saturating. Every task went through several versions. Our first versions gave 1.00 to almost every model, which tells you nothing. For example, Knowing when to ask went from v1 (1.00 for all) → v2 (0.63–0.96) → v4. Bug fix v2 put every bug away from where the symptom appears, which spread scores from 0.28 to 0.89.
• Controls catch over-correction. Some items are already correct or already complete, so a model that changes everything, or asks about everything, loses points.
• Measured, not guessed. Memory answers come from tracemalloc, and complexity "slow" versions are timed. Our grader refuses an item if its slow version isn't actually slow.
• We fixed our own grader bugs when a model's answer was fair and our key was wrong.

The models (15)

We picked a lineup to answer three questions: Does size matter? Does the provider matter? Can open models compete?

• Closed, 3 tiers per family: Claude Opus / Sonnet / Haiku 5.5 · GPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna, GPT-5.6 Terra · Gemini 3.1 Pro, 3.8 Flash, 3.7 Flash · Grok 4.20 Reasoning
• Open-weight: Gemma 4 31B, GLM-5, DeepSeek R1, Qwen3 Next Instruct
• All models ran in their default mode, the way most developers would use them.

Grok's other two tasks didn't finish because of heavy load. The Kaggle chart counts them as 0 (about 22%), so please ignore its overall score.

Beyond the overall score

1. No model is good at everything. Opus 5.5 leads overall, but nobody scored 1.00 on bug fix or memory reasoning. The overall score hides very different profiles.

2. The most surprising result: Astra. GPT-6 Astra was the best model in the whole benchmark at memory reasoning (0.92). It was one of the worst at complexity analysis (0.43), along with GPT-6.1 Sol. Understanding memory and understanding time complexity are supposed to be closely related skills, but here they behaved like separate ones. Looking at its raw answers explained it: Astra said "already optimal" on all 14 items. Every time, the best possible bound it gave was the same as the current code's. So it got the 6 items that really were already O(n) right and failed all 8 that could be made faster, for example loops that are O(n²) but could be O(n). On one item it even refused to give a bound because the loop never terminates when size=0. It treated "keep the exact current behavior, edge cases included" as more important than the question we actually asked. Astra wasn't bad at reading code. It was too conservative to suggest a faster version. A model can be very accurate at describing what code does and still be bad at saying "this could be faster."

3. Open models can compete, on some tasks. Gemma 4 31B, a model you can run yourself, scored 1.00 on complexity and 0.78 on bug fix. On bug fix that ties Gemini 3.1 Pro and beats GPT-6.1 Sol, GPT-6 Luna and both Gemini Flash models. Its overall score (0.70) matches GPT-6.1 Sol, and its whole benchmark run cost about $0.16. GLM-5 also got 1.00 on complexity. The open models' weak spot is memory reasoning: GLM 0.38, Gemma 0.31, DeepSeek 0.23, Qwen 0.08.

4. Memory reasoning separates models the most (0.08 → 0.92). The most common mistake was counting a sort copy twice. On the presorted item, several models were off by exactly 2×.

5. Models fix the file the bug report points to. In bug fix, models handled bugs located where the symptom appeared, but missed bugs that live elsewhere. Finding the right file, not writing the patch, is the real skill being tested.

6. Models guess instead of asking. On Knowing when to ask, models were rarely too cautious. Their failures were confident guesses whenever a tempting default existed.

7. Bigger isn't always better, and expensive isn't always better. Haiku 5.5, the smallest Claude, tied Opus on memory (0.85) and beat Luna and Astra overall. On the cost chart, GPT-6 Luna is the most efficient model (about 79% for under $0.10). Gemini 3.1 Pro was the most expensive and still didn't finish first.

8. Complexity is our easiest task. Six models scored 1.00 even after we removed hints and buried costs 2–3 files deep. We're reporting that as a limitation rather than hiding it.

What this changed for us

We expected a ranking from "best model" to "worst model". Instead, every model had its own profile. If you're choosing a model for maintenance work, the overall score is the wrong number to look at. A model that's best at memory can be one of the worst at complexity, and a cheap open model can match a flagship at finding bugs. Test models on the skill you actually need.

What we'd measure next

• A harder complexity task, since it's still saturated at the top
• Several runs per model, because close models changed score between runs
• Agentic mode, where models explore the repo with tools instead of reading it all in context
• More repos and more languages (JavaScript, a legacy C# or PHP codebase)

Limitations

One repo, Python only, one run per model, and Grok is incomplete. The total cost of all runs was about $23.

Try it yourself

• Benchmark: https://www.kaggle.com/benchmarks/iseoluwaolowogoke/legacyshop-can-an-ai-maintain-a-real-codebase
• Dataset (the repo): https://www.kaggle.com/datasets/iseoluwaolowogoke/legacy-shop-repo-v1
• Each task page has Compare Outputs, so you can read every model's raw answer.

Thanks to Kaggle for the Benchmarks platform. Built by @g00ds0ul and @joyy-dev.