Email tests rarely fail because an email is completely absent. More often, the test sees the wrong message, reads the inbox too early, or loses the evidence that would explain what happened. A temp mail generator can provide a useful isolated fixture, but isolation alone does not make an end-to-end test diagnosable.
This is the workflow I use when a Playwright email test is flaky: define the expected states, capture the first useful evidence, and make every retry explain what changed. The goal is not to add more waits. It is to make the failure tell us which boundary broke.
The failure is often a missing state, not a missing email
Consider a signup test that requests a verification code. The test opens an inbox and waits for a message with the subject âVerify your account.â It passes locally, then fails in CI with âmessage not found.â
That message hides several different failures:
- The signup request never completed.
- The message was sent to a previous fixture address.
- The inbox API returned an older message first.
- The email arrived after the polling deadline.
- The code was present, but the selector matched the wrong element.
- The test found the message but failed while parsing its body.
Each case needs a different fix. A longer timeout only makes some of them slower. The first diagnostic step is to record the boundary events: fixture created, address submitted, request accepted, message observed, code extracted, and verification completed.
Model the fixture as a state ledger
I prefer a small state ledger over a single waitForEmail() helper. The ledger gives each transition a timestamp and a compact detail value without storing the whole mailbox contents in every log line.
type EmailState =
| "fixture-created"
| "address-submitted"
| "message-observed"
| "code-extracted"
| "verified"
| "expired";
const states: Array<{ name: EmailState; at: string; detail?: string }> = [];
function mark(name: EmailState, detail?: string) {
states.push({ name, at: new Date().toISOString(), detail });
}
The ledger should be attached to the test result when a step fails. Do not log the verification code itself; record its source message ID, subject, and age instead. This keeps the artifact useful without turning a CI report into a copy of the inbox.
A state ledger also catches an easy-to-miss problem: the first clue are often in the order of events. If message-observed occurs before address-submitted, the test may be reading a stale message or sharing a fixture across workers.
Build a diagnostic Playwright helper
The helper should poll for a message using a stable identity, not a broad text match. A good identity usually combines the fixture address, a test-run token, and a bounded age. Subject text can be part of the check, but it should not be the only check.
async function waitForVerification(page: Page, token: string) {
const deadline = Date.now() + 30_000;
while (Date.now() < deadline) {
const message = await readLatestMessage();
if (message && message.runToken === token) {
mark("message-observed", message.id);
const code = extractCode(message.text);
if (!code) throw new Error(`Message ${message.id} has no code`);
mark("code-extracted", message.id);
return code;
}
await page.waitForTimeout(1_000);
}
mark("expired", `token=${token}`);
throw new Error("Verification message did not arrive before the deadline");
}
In the real helper, keep the polling interval and deadline configurable. Capture one sanitized response summary on timeout: message count, newest message timestamp, and the requested token. That is usually enough to tell whether delivery or matching failed.
If a selector is involved, prefer a role or a data attribute that the application owns. Avoid a selector such as text=123456; another message can contain the same number. Playwright is very good at reporting selector failures, but only when the test gives it a stable target.
Keep the mailbox boundary explicit
The mailbox is an external dependency even when it is created by a test service. Give each worker a unique address or run token, and clean it up after the test. Also separate the browser timeout from the email delivery timeout. They measure different things and should produce different errors.
When documenting fixtures, search terms can be messy. Someone may write tepm mail com or temp gamil com in a failure ticket. Keep those phrases in troubleshooting notes if they help search older incidents, but do not turn them into selectors or production validation rules. The phrase temp mail mail can likewise appear in an imported test description without being a reliable fixture identifier.
For privacy-sensitive flows, review what the fixture service retains and who can access it. These privacy notes for email checks are a useful reminder that test data still deserves a boundary and a deletion policy.
A repeatable CI failure checklist
When the test fails, check the evidence in this order:
- Identity: Was the same address or run token used from setup through message lookup?
- Application event: Did the signup request return the expected status before polling began?
- Delivery: Did any message arrive within the deadline, even if it had an unexpected subject?
- Matching: Did the message contain the current token, or was it stale?
- Parsing: Was the message body format different from the local fixture?
- Browser state: Did the page navigate, reload, or lose the verification form while waiting?
- Artifact quality: Can another engineer replay the relevant inputs from the CI artifact?
If the failure is intermittent, save a small replay pack with the state ledger, sanitized message metadata, request correlation ID, and the final page screenshot. An approach based on replay packs for easier debugging works well because it preserves the facts without asking a developer to guess which retry was meaningful.
Questions that help classify the failure
Should I increase the email timeout first?
No. First determine whether the message arrived late, arrived under a different identity, or was never requested. Increase the deadline only when delivery latency is the known cause, and keep the failure artifact so the new budget remains visible.
Is a temp mail generator suitable for every test?
It is useful for isolated signup and verification scenarios, especially when each run gets a unique fixture. It is less suitable for tests that need long-term mailbox history, guaranteed retention, or sensitive production-like data. In those cases, a controlled fake mail adapter may be more predictable.
What makes an email test reliable?
Reliable tests have unique identity, explicit states, bounded polling, stable selectors, sanitized evidence, and cleanup. They also fail at the first broken boundary. That last part matters most: a test that says âverification failedâ is hard to repair, while one that says âmessage arrived for the previous run tokenâ points directly to fixture isolation.
The practical rule is simple: treat email as a sequence of observable states, not a single asynchronous event. Once Playwright records that sequence, flaky failures become smaller QA problems with evidence instead of mysterious red builds.
Top comments (0)