system deployments: cancel older deployments even if inactive

[tgross]

In Nomad 1.11.0 we added deployments for system jobs. As part of that work we added the logic to "cancel" deployments that belong to older versions of the job (in #26953). This involves two operations: move the reconciler state tracking that deployment to an "old" field, and optionally recording that we'll want to update the state store as part of the plan.

This logic was copied from the cluster reconciler used for service/batch jobs, but the clause that updates the reconciler state was placed inside the clause that updated the state store. This prevents us from creating new deployments for updated versions of the job unless the prior version is successful or still active.

Move this logic outside the Active() check, and add test coverage we were missing of this code path.

Fixes: #28538
Ref: https://hashicorp.atlassian.net/browse/NMD-1735
Ref: #26953

Testing & Reproduction steps

Using the following jobspec derived from the ones posted by @daniel1302
in #28538 (comment)

jobspec

variable "version" {
  type = string
}

variable "path" {
  type    = string
  default = "/"
}


job "example" {
  type = "system"

  update {
    max_parallel      = 1
    healthy_deadline  = "30s"
    progress_deadline = "1m"
  }

  group "nginx" {
    network {
      port "http" {
        to = 80
      }
    }

    task "nginx" {
      driver = "docker"

      config {
        image = "nginx:1.28-alpine"
        ports = ["http"]
      }

      env {
        VERSION = var.version
      }

      service {
        name     = "nginx"
        provider = "nomad"
        port     = "http"

        check {
          type     = "http"
          path     = var.path
          interval = "2s"
          timeout  = "1s"
        }
      }

      resources {
        cpu    = 50
        memory = 128
      }
    }
  }
}

Run the following and wait for the deployment to be successful.

$ nomad job run -var version=1 -var path=/ ./example.nomad.hcl

Run it again with a broken path, and wait for the deployment to fail.

$ nomad job run -var version=2 -var path=/broken ./example.nomad.hcl

$ nomad deployment status
ID        Job ID   Job Version  Status      Description
27dc253d  example  1            failed      Failed due to unhealthy allocations
da57d6e7  example  0            successful  Deployment completed successfully

Run it a third time and you'll see there's no deployment created. With this patch we do see a deployment as expected.

$ nomad job run -var version=3 -var path=/ ./example.nomad.hcl

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
• Documentation n/a
• LLM Usage If an LLM was used to generate any code, please ensure and confirm you have read
  and followed the AI usage guide.

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

• If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.