Where Did My Data Go? Benchmarking LLMs as Data Pipeline Detectives

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

Every data engineer has gotten this ticket at least once. Someone from sales writes that Umbrella Corp is right there in the CRM but missing from the dashboard, and could you please check where it went, ideally by the end of the day.

Nobody knows why, that's the whole point of the ticket. Between the CRM and the dashboard there is a pipeline that ingests the record, extracts the company name, merges duplicates and filters some of them out, and the record could have been lost at any of these steps. So you go digging through the logs, half of which were rotated out last Tuesday, and there in the final table you find "Umbrella Corporation Inc". Is that our Umbrella? Maybe. Probably? It's the same name, give or take an Inc.

A good investigator says "it was filtered out, here is the log row" or "I can't tell" when the trail has gone cold. A bad one goes with "probably", picks the lookalike, closes the ticket and goes to lunch, and two weeks later someone finds out the dashboard has been showing numbers for a completely different company.

Now everyone wants to hand these tickets to an LLM, and any LLM can spot a lookalike in the final table. But will it notice when the logs simply aren't enough to say where the record went? To find out I built Where Did My Data Go?, it works like this:

1. I simulated 15 small pipeline scenarios, and in each one I decide what really happened to one company record: it was never loaded, its name couldn't be read, a rule filtered it out, it got merged into the wrong company or it survived under a different id.
2. The pipeline writes its logs the way a real one would, and then I delete some of them, sometimes nothing, sometimes a few, sometimes almost all of them.
3. The model gets whatever logs are left and has to say what happened to the record or "can't tell" if the logs that are left don't prove any single answer.
4. Plain code checks the answer, there's no LLM judge. It checks against what the remaining logs actually prove, not against the hidden truth, so if the proof was deleted, "can't tell" is the right answer even when a guess would have been lucky.

So every answer is one of six verdicts:

There's usually a lookalike sitting in the final table, like our "Umbrella Corporation Inc", and I even have a pair of twin cases with exactly the same logs, where in one the lookalike really is our company and in the other it's a different company that just has a similar name, so the only honest answer to both is "can't tell".

Every investigation is asked in two modes, as a plain question and with a short rulebook that says don't guess and point at the exact log row that proves your answer, to see how much a few lines of instructions change.

Models Tested

I picked models the way I would for a real pipeline. This check would run on every ticket every night, so realistically nobody puts the most expensive model in that loop. That means the cheap tier of each lab, two open-weights models you can host yourself and Claude Sonnet 5, the model I built the benchmark on, as a pricier reference:

Each model did 107 investigations per track through Kaggle's Model Proxy, and none of the calls failed.

Findings

The 15 cases come in 33 versions with more or fewer logs, and each version is asked 3 times (5 for the trickiest ones), which makes 107 investigations per track, 44 of them cold cases where "can't tell" is the only right answer. I look at three things: how often a model guesses on those cold cases, how often its full answer (verdict and ids) is correct and how often it's correct and proven, meaning that whenever it claims something, the log rows it cites actually exist and prove it. And these are three different things, a model that says "can't tell" to everything would never guess and still score only 44%. That's not 44 out of 107 by the way, every score here is averaged per case, so each of the 15 cases counts the same no matter how many versions it has.

Here is everything sorted by price:

In this lineup price didn't buy caution, in either direction. The cheapest model, GPT-6 Luna, never guessed once and got everything right on instinct for 18 cents per thousand investigations, while Sonnet guessed on almost every third cold case and Kaggle's default Gemini 3.7 Flash cost almost 40 times more than Luna and still guessed. The two worst guessers, Qwen and Flash-Lite, are cheap too, so the price tag just tells you nothing, you have to measure it. Speed doesn't follow quality either: Flash-Lite answers in under a second and guesses on almost half the cold cases, Luna and Sol take about two seconds and are almost always right and Gemma needed around 40 seconds per investigation on Kaggle's hosting.

A few paragraphs of rules in the prompt took seven of the eight models to zero guessing and Qwen from 59% to 7%, and with them Haiku and Sonnet end up with exactly the same score, except Haiku is 14 times cheaper. Asking for the exact log row matters too, when I tried the rules without it during development Sonnet still guessed 4.5% of the time. If I had to put a model on the ticket queue tomorrow, it would be a cheap one with the rules, not an expensive one without them.

My favorite failures are the ones humans make too. In one case two different companies with different registry numbers get merged by mistake and then a review rule throws the merged entity out. Gemini 3.7 Flash and Flash-Lite blamed the filter instead of the bad merge every single time, with and without the rulebook, and Gemma did the same once it had the rulebook, which is what you get when you read the logs from the bottom and stop at the first error. Qwen went with "if it's not in the log, it never happened". The record wasn't in an ingestion log that was clearly marked as partial, so it decided the record was never loaded, even with "in a partial log a missing row proves nothing" written right there in the prompt. And not guessing isn't the same as being useful. Sol never guessed, but on instinct it sometimes said "I can't tell" even on a plain registry-number match, and every "I can't tell" on an easy ticket lands back on your desk.

I built the whole benchmark on Sonnet and was pretty sure it would be the careful one but on instinct it guessed on 25% of cold cases during development and 30% on Kaggle, on the same traps, while the cheapest model didn't guess at all. And Gemma somehow did better without the rulebook than with it. Next I'd like to let the model ask for the logs it needs instead of getting them all at once, and see if it goes looking for the merge log before blaming the filter.

My Benchmark

Kaggle benchmark: https://www.kaggle.com/benchmarks/olgazju/where-did-my-data-go

Code: https://github.com/olgazju/where_did_my_data_go_benchmark

So next time a "the customer is missing from the dashboard" ticket comes in, I know the rulebook goes into the prompt no matter which model I use, and the model doesn't have to be expensive to say "I can't tell" instead of closing it with the wrong Umbrella.

Case closed. Probably.