scheduler: keep draining batch alloc counted when node is re-enabled

[szabolcsgelencser]

Description

When a node that is currently draining a batch job allocation is re-enabled
(set back to eligible) before the allocation has finished stopping, Nomad
creates a new replacement allocation for the batch job. This contradicts the
documented batch-drain behavior, which says a batch allocation should be
allowed to complete on the draining node and should not be replaced.

This appears to be a regression / follow-up edge case to the fix shipped in
#26961, which corrected the case where a new allocation was created
immediately on batch-job drain. That fix works correctly when the drain runs
to completion. The bug shows up only in the narrow window where the
allocation is still running (i.e. it takes some time to shut down) and the
operator toggles the node back to eligible during that window.

The evaluation that creates the extra allocation is shown in the UI as
triggered by node-update, which lines up with the node going from
ineligible back to eligible.

Testing & Reproduction steps

1. Submit a batch job with a task whose graceful-shutdown takes a noticeable
   amount of time (e.g. ~1 minute — a kill_timeout long enough that the
   alloc remains in running for a while after being asked to stop).
2. Wait for the allocation to be running on some node.
3. Drain the node. The drain begins; the
   allocation is marked for stop but is still running on the node (this is
   the documented behavior — the batch alloc is allowed to finish).
4. Before the allocation actually transitions to a terminal client status,
   re-enable the node.
5. Observe the job's allocations and evaluations.

Links

Fixes #28008

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
• Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
  and job configuration, please update the Nomad product documentation, which is stored in the
  web-unified-docs repo. Refer to the web-unified-docs contributor guide for docs guidelines.
  Please also consider whether the change requires notes within the upgrade
  guide. If you would like help with the docs, tag the nomad-docs team in this PR.

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

[hashicorp-cla-app]

All committers have signed the CLA.

[hashicorp-cla-app]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

_{Have you signed the CLA already but the status is still pending? Recheck it.}

[tgross]

Marking this as "request changes" for now just until we get the details of the bug worked out with you over in #28008 (comment)

[tgross]

As per our discussion in #28008 (comment) this LGTM.

Can I get you to run make cl to add a user-facing changelog entry? Thanks!

[szabolcsgelencser]

As per our discussion in #28008 (comment) this LGTM.

Can I get you to run make cl to add a user-facing changelog entry? Thanks!

Sure thing! ✅ 🙂

[szabolcsgelencser]

@tgross would you like to see any other changes?

[tgross]

LGTM. Thanks @szabolcsgelencser!