The most common way a verification lies is not by erroring: it is by going green.
These four checks cost seconds and prevent the “tested, all good” that tested nothing.
1. A 200 is not a result
An anti-bot page, a consent wall and a login redirect all return 200. Before parsing,
assert a marker you expect to see (the product name, a word from the title). If the marker is not
there, the body belongs to another page.
2. The count has to add up
The tool says total: 5 and you parsed 3. Do not report the 3 you have: re-fetch and
parse in code. A report with fewer items than reality is worse than no report, because it looks
complete.
3. Same host, another app
A hostname can point at two different applications (the last rule wins, with no error). Validate
by comparing the response body, not just the status: another app's endpoint can answer 200
in place of the one you expected.
4. Unverifiable is not verified
A command that does not exist, a service that is down, a full disk, a stuck machine: the absence
of proof is never proof of success. Write “unverifiable” and the reason, in one line —
instead of letting it look like a pass.
In a gate that becomes three states, not two: passed, failed (the only one that is
yours) and unverifiable (the machine's). Collapsing the third into “passed” makes the gate
approve without proving; collapsing it into “failed” makes the gate stop enforcing.
The test that catches this
Write the check so it fails when the verification does not happen — not one that passes by
omission. A check that only fails when the command runs and errors is measuring the command, not
the result.
The Agent That Does Not Lie — R$ 37, the 14 chapters on telling a real 200 from a decorative one: o-agente-que-nao-mente — R$ 37, instant delivery.
Next in this series (the one after this): https://dev.to/samuelfmedeiros/the-agent-said-done-and-produced-nothing-verifying-is-not-producing-59hn
See the AI Test Loop skill
See all skills
Want this wired into your project? Guided setup from
R$ 1.500, with the first cycle run together.
Top comments (0)