Dấu or Die: Can LLMs put the accents back on Vietnamese?

I built Dấu or Die, a Kaggle benchmark that removes the diacritics from Vietnamese sentences and asks LLMs to put them back. Each missing tone mark can change a word's meaning completely.

Benchmark: https://www.kaggle.com/benchmarks/tuanfelixnguyen/du-or-die-vietnamese-diacritic-restoration
Task & code: https://www.kaggle.com/benchmarks/tasks/tuanfelixnguyen/dau-or-die-vietnamese-diacritics

Why this task

Vietnamese has six tones plus vowel marks (ă, â, ê, ô, ơ, ư, đ). Millions of us still type without them ("không dấu"):

• on phones without a Vietnamese keyboard,
• in URLs, file names and usernames,
• in old databases and SMS systems that only accept ASCII.

Remove the marks and many words collapse into the same string. ban could be bàn (table), bán (sell), bạn (friend) or bận (busy). Native speakers fix this from context without noticing. I wanted to know whether LLMs can do the same, or whether they just default to the most common word.

My favourite test sentence:

Cai ban nay ban bao nhieu tien vay ban?
→ Cái bàn này bán bao nhiêu tiền vậy bạn? ("How much is this table, friend?")

The same three letters appear three times, and each one must become a different word.

How the benchmark works

I wrote 37 sentences by hand, in 5 categories:

Then for each sentence:

1. The diacritics are stripped automatically with Unicode NFD normalisation, and đ is mapped to d.
2. The model is asked to restore the text and must return structured output {restored_text: str}.
3. Scoring is fully deterministic, with no LLM judge:
   • Headline score: the share of sentences restored exactly. One wrong tone counts as wrong, because one wrong tone can change the meaning.
   • Also reported: word-level accuracy per category.

Built with the kaggle-benchmarks library: one @kbench.task per sentence, aggregated by a parent task via .evaluate().

Models tested

I picked seven models from five different labs, mixing "fast/cheap" tiers with larger models. I wanted to see whether size or lab matters more for a low-resource-ish language task.

I also added Gemma 4 26B, Google's open-weights model, but its run ended in an error before it produced a score. My scoring code assumes at least one sentence comes back in the requested structured format, so a model that fails that on every sample crashes the aggregation. I ran out of time to diagnose it before the deadline. That is a bug in my harness, not a verdict on Gemma.

Findings

1. A newer "Flash" model beat an older "Pro" model. Gemini 3.7 Flash restored every sentence, while Gemini 2.5 Pro missed 3. For this task, a newer generation mattered more than the bigger tier.

2. Being big doesn't help if the model is specialised for something else. Qwen 3 Coder has 480B parameters, by far the largest model here, yet it came last with 25/37. A model tuned for code seems to have lost some fine-grained, multilingual detail. Vietnamese tone marks are exactly that kind of detail.

3. The spread is wide even though the task looks trivial. Every native speaker restores these sentences without thinking. Yet the gap between the best and worst model is 12 sentences out of 37. One wrong tone counts as a wrong sentence, and that is how a reader experiences it too: "bán nhà" (sell the house) and "bàn nhà" (table house) are not "almost the same".

4. Cost buys very little at the top. Kaggle's score-vs-cost chart shows that GPT-5.4 mini reaches 0.89 for roughly $0.02 for the whole benchmark. GLM-5 gains only one more sentence (0.92) but costs over 10× more, around $0.3. Claude Haiku (≈$0.01) and Qwen Coder (<$0.01) are cheaper, but they lose accuracy quickly. For a production "add the accents back" feature, a small model is the sensible choice, with one condition: it has to be the right small model.

What surprised me

The benchmark found its own ambiguity. On my first development run, Gemini 3.7 Flash restored Sep bao hop online… as "Sếp báo họp online" ("the boss announced a meeting") instead of my reference "Sếp bảo họp online" ("the boss said to meet"). Both are correct Vietnamese. When I re-ran the same model at temperature 0 for the leaderboard, it chose "bảo" and scored a perfect 1.00.

Two lessons:

• Removing diacritics sometimes destroys information that no amount of context can recover, so exact match is a slightly harsh metric. A future version should accept a set of valid answers per sentence.
• Model outputs are not fully stable, even at temperature 0, and the wobble shows up exactly on the genuinely ambiguous items. A single leaderboard number hides that.

What I'd measure next

• Fix the harness so format failures count as 0 instead of crashing, then re-run Gemma and other open-weights models
• Multiple valid answers per sentence (bảo/báo), plus repeated runs to measure how stable each model is
• Teencode / slang ("ko", "dc", "j z") mixed with missing accents
• Longer paragraphs, where context can help more but errors also add up
• Regional vocabulary (Northern vs Southern words)
• The reverse direction: can a model detect a wrong diacritic that a typo introduced?

Credits

Solo entry by Tuan Nguyen. Built with kaggle-benchmarks. The sentences and the scoring are my own. Feel free to fork the task and add more Vietnamese sentences!