DEV Community

Cover image for A job that could never succeed: skipping note emails for deleted recipients
Priyanshu Jha
Priyanshu Jha

Posted on

A job that could never succeed: skipping note emails for deleted recipients

Some bugs don't fail for the user who is still around. They fail for a user who is already gone, in a background job that cannot possibly succeed, and they leave a trail of retries in the dead set.

This is the story of gitlab-org/gitlab!268706. It stops GitLab from trying to send a note notification after the recipient has been deleted, and it went through a review that made both the guard and the specs stricter. I'll cover the issue, the first version, and the two review comments that changed it.

The issue: the same failure, on the other lookup

The issue was gitlab-org/gitlab#638487, opened by Danny Bailey. It is a follow-up, and that matters.

An earlier MR, !267804, had fixed the same shape of failure for deleted notes. Emails::Notes#setup_note_mail used to load the note with a find that raises when the record is missing. If the note was deleted after a Notify#note_*_email job was queued but before the job ran, ActiveRecord::RecordNotFound was raised, ActionMailer::MailDeliveryJob retried it, and the job ended in the dead set. The fix was to load the note with Note.find_by_id and return false when it was missing, so the action returned before calling mail.

The recipient lookup was left out of scope. It still used User.find(recipient_id). If the recipient was deleted in that same window, the job took the same path: raise, retry, dead set, and it could never succeed.

Danny's proposal was to mirror the note guard: look the recipient up with User.find_by_id, and skip delivery when it is missing.

From a queued note email to the dead set, and the skip after the fix

The plan

I commented on the issue with this plan:

I plan to mirror the deleted-note guard from !267804: load the recipient with User.find_by_id in Emails::Notes#setup_note_mail and return early when missing, with a focused spec.

Danny assigned me the same day: "Sure thing, I've assigned you, go for it." Matching an existing, just-merged pattern in the same method is the easiest kind of change to review, as long as the new guard really behaves like the old one.

Version 1: the same method, one mailer

The production change in the first commit (d89428c0) was two lines in setup_note_mail. After the note is loaded and the project and group are derived, the recipient is no longer loaded with find:

@recipient = User.find_by_id(recipient_id)
return false unless @recipient
Enter fullscreen mode Exit fullscreen mode

If that returns false, every note email action returns before calling mail, and SentNotification.record_note is never reached. The YARD comment on the method now says it returns false when the note or the recipient was deleted before the job ran.

The first spec covered note_issue_email only. It used a non_existing_record_id as the recipient, shared the 'no email is sent' example, and asserted that deliver_now did not create a SentNotification:

context 'when the recipient is deleted before the email is sent' do
  let_it_be(:note) { create(:note_on_issue, noteable: issue, project: project) }

  subject { described_class.note_issue_email(non_existing_record_id, note.id) }

  it_behaves_like 'no email is sent'

  it 'does not record a sent notification' do
    expect { subject.deliver_now }.not_to change { SentNotification.count }
  end
end
Enter fullscreen mode Exit fullscreen mode

One small workflow bump: my first @gitlab-bot ready ran before GitLab had indexed the diff, so no reviewer was assigned. I posted ready again once the change was visible, and the pipeline was running.

The review: return earlier, and test the path that actually failed

Danny left two non-blocking suggestions, both of which made the change better, and I applied both in 82c370a6.

1. Don't derive a project for a mail you will not send

The first version looked up the recipient after setting @project and @group. Danny's nitpick:

consider looking up the recipient right after the note guard, so we bail out before deriving @project/@group when there's nothing to send.

That's a small move with a clear reason. If the recipient is gone, there is no mail, and there should be no SentNotification. Doing the extra work first doesn't break anything, but the method reads as "set up the mail, then maybe notice we can't send it." The final order is note, then recipient, then project and group:

def setup_note_mail(note_id, recipient_id)
  @note = note_id.is_a?(Note) ? note_id : Note.find_by_id(note_id)
  return false unless @note

  @recipient = User.find_by_id(recipient_id)
  return false unless @recipient

  @project = @note.project
  @group = @note.noteable.try(:group)
  @group ||= @note.noteable.resource_parent if @note.noteable.try(:resource_parent).is_a?(Group)

  @sent_notification = SentNotification.record_note(@note, recipient_id) if (@project || @group) && @note.persisted?
Enter fullscreen mode Exit fullscreen mode

2. Cover every note mailer, through deliver_later

The issue's failure is not deliver_now. It is a job that raises inside ActionMailer::MailDeliveryJob, is retried, and lands in the dead set. Danny pointed at a shared example that already existed for deleted notes:

could this cover all six note mailers and use 'a note email for a deleted note', like the deleted-note context just above? That shared example runs the job through deliver_later and asserts it completes without raising, which is the failure from #638487.

My first spec used 'no email is sent' and only note_issue_email. That shows no mail goes out on the synchronous path. It does not show that the job completes without raising, and it does not show that note_commit_email, note_merge_request_email, note_snippet_email, note_design_email, and note_wiki_page_email take the same early return. Those six methods all go through setup_note_mail, but a regression in one of them is exactly the kind of thing a parameterized example is for.

The final spec runs all six through the shared example, keeps the SentNotification check, and adds the same case for note_epic_email in ee/spec/mailers/notify_spec.rb:

context 'when the recipient is deleted before the email is sent' do
  let_it_be(:note) { create(:note_on_issue, noteable: issue, project: project) }

  where(:mailer_method) do
    %i[
      note_commit_email
      note_issue_email
      note_merge_request_email
      note_snippet_email
      note_design_email
      note_wiki_page_email
    ]
  end

  with_them do
    subject { described_class.public_send(mailer_method, non_existing_record_id, note.id) }

    it_behaves_like 'a note email for a deleted note'

    it 'does not record a sent notification' do
      expect { subject.deliver_now }.not_to change { SentNotification.count }
    end
  end
end
Enter fullscreen mode Exit fullscreen mode

Danny reviewed the new version, approved it, and asked Serena Fang for a second backend look. Serena approved it and set merge when pipeline succeeds. The MR merged on 10 November 2026.

Review: an earlier return, and every note mailer through deliver_later

What I took away

1. When a just-merged MR leaves a lookup out of scope, that's often the next issue. The deleted-note guard was the template. I didn't need a new design. I needed to apply the same pattern to the line that was skipped.

2. find and find_by_id are not interchangeable in jobs. find raises RecordNotFound. A mail job treats that as a transient failure and retries it. If the record is gone for good, the retries are wasted work and the dead set is noise. find_by_id returns nil, and you can leave quietly.

3. Put the guard before the work, not after. Looking up the recipient after deriving @project and @group was correct, but it wasn't kind to the next reader. If you're going to return, return first.

4. Test the failure mode from the issue, not a convenient stand-in. 'no email is sent' plus deliver_now is weaker than 'a note email for a deleted note' plus deliver_later. The bug was a job that raised. The spec should be a job that doesn't.

5. Parameterize across every entry point that shares the guard. Six FOSS mailers and the EE epic mailer all call setup_note_mail. One example on note_issue_email would have left the rest to faith.

Links

Thanks to Danny Bailey for the issue, for pointing at the shared example I should have used first, and for the early-return nudge, and to Serena Fang for the maintainer review.

If you've been thinking about contributing to GitLab, the community issues are a great place to start: contributors.gitlab.com.

Top comments (0)