A monitoring script is supposed to stop shouting once you've been told. Mine shouted every morning for eight days, and the reason was a file with zero bytes.
The design that looked fine
The pattern is common: a cron job runs every hour, checks something, and when the check fails it sends a message. To avoid a message every hour, it keeps a latch — a small JSON file with the date of the last time it warned. If the latch says "already warned today", the job stays quiet.
def already_warned(chave):
try:
with open(LATCH) as fh:
dados = json.load(fh)
return dados.get(chave) == today()
except FileNotFoundError:
return False
except Exception:
return False # <-- the line that cost me 8 days
Note the last branch. It exists "to avoid crashing the monitor". It does something else entirely: it converts every imaginable bug into "not warned yet", which is the loudest possible state.
What actually broke
The write side looked like this:
def grava_latch(chave):
dados = {}
try:
with open(LATCH) as fh:
dados = json.load(fh)
except Exception:
pass
dados[chave] = time.strftime("%Y-%m-%d")
with open(LATCH, "w") as fh:
json.dump(dados, fh)
time was never imported in that module. NameError: name 'time' is not defined — raised inside the try, caught by the bare except Exception: pass. Except no: the failing line is after the try, so the exception propagated to the caller, which wrapped the call in its own except: pass, then sys.exit(0). The file had already been created by open(LATCH, "w") before the crash. Result: a latch that exists, is 0 bytes, and reads as "never warned".
Every symptom pointed the wrong way:
- The job's exit code was 0.
- The job's output was empty.
- The latch file existed with a fresh mtime.
Three green signals, one broken behaviour. If I had only looked at systemctl status or the job's status field, I would have concluded the alert was a duplicate-delivery problem and gone hunting in the message sender.
The three rules I extracted
1. A latch is state, so verify it is state. A zero-byte, unparsable or stale file must be treated as an error, not as "nothing recorded yet". After writing, read it back and assert the content:
with open(LATCH, "w") as fh:
json.dump(dados, fh)
fh.flush()
os.fsync(fh.fileno())
assert os.path.getsize(LATCH) > 0
Cheap, boring, and it turns a silent 8-day failure into a single loud crash on the first run.
2. except Exception: pass is a decision to lose the bug. Keep the catch, but surface the exception when a debug flag is set, or at minimum write repr(exc) into a sibling .err file. Silence belongs to cases you can name (FileNotFoundError on first run), not to the case "I do not know what happened".
3. Status fields lie in one specific direction. A job that exits 0 without doing anything is indistinguishable from a job that did it. This is the same class of failure as an HTTP 200 from the wrong server: the indicator measures liveness, not the thing you care about. Assert the effect, not the exit.
Why I keep finding these
Every one of these bugs is a verification bug wearing a different costume. I have now hit, in the same codebase, in a few weeks:
- A parser whose regex matched a single digit, so
1.234silently became1and a dashboard read the same number for three weeks. - A guard script reading the wrong database file, reporting
0 pending— accurate for a database that no longer received events. - A watchdog that restarted services on boot because
systemctl is-activereturns a non-zero code foractivating, and the shell appended a second line that never matched. - This one: a flag file that existed, empty, and therefore said "nothing happened".
The shape is identical each time: the check reports on a proxy, the proxy drifts from reality, and the drift is invisible because the report is green. You do not fix this class by being more careful. You fix it by making each check answer a narrower question with evidence attached: not "is the service up" but "which PID is listening and does its identity match the version I deployed"; not "did the job run" but "did the effect it was supposed to produce appear".
The part that actually costs the time
Not the bug. Bugs are cheap. The expensive part is the two days spent looking at the alerting pipeline, because the alert had to be lying — the job was green.
The rule I follow now, before touching any fix: when something contradicts itself, the first job is to find which measurement is fake. One tail of the raw output, one read-back of the file, one query of the actual store — done in that order, before any hypothesis about the system's behaviour. Reading the raw tail instead of grepping for the success string is what surfaced this one; the traceback was in the last three lines, and the grep for warned had hidden it.
If you want the checklist I use to make "done" mean something before shipping an agent or a job that claims it worked, it is here as a free package:
The latch and the watchdog are two of the seven chapters in The Agent That Delivers: guia-agente-que-entrega — R$ 67, instant delivery.
Next in this series (the one after this): https://dev.to/samuelfmedeiros/five-open-issues-one-bug-your-mcp-oauth-refresh-token-is-being-replayed-after-rotation-5705
It is the gate I put in front of my own agents: separate reviewer, no paid model in the loop, exit codes that CI can read, and never a pass for an inconclusive run.
Top comments (0)