Skip to content

Handle "Unexpected message id" errors after hard pod shutdown #267

Closed
vaadin/flow
#23267

Description

@mlopezFC

We are investigating issues caused by "hard pod shutdowns" (e.g., OOMKilled or node failures where SIGKILL is sent). When a pod is terminated abruptly, the standard graceful shutdown hooks do not run.
If this termination occurs during or before the latest UI changes are serialized to Hazelcast, the distributed session state becomes stale or incomplete. When the client attempts to reconnect to a new healthy pod, the client-side state (UI message ID) is ahead of the server-side state (deserialized from Hazelcast).
This results in the server throwing an Unexpected message id from the client error, which currently manifests as a broken UI for the end user.

Current Behavior

  1. Pod A processes a user interaction but is SIGKILLed before it can serialize the updated UI state to Hazelcast.
  2. The Client detects the connection loss and automatically reconnects.
  3. Kubernetes routes the request to Pod B.
  4. Pod B retrieves the session from Hazelcast, but this session data is stale (missing the latest state).
  5. The Client sends a request with Message ID: X.
  6. Pod B expects Message ID: X-1 (or earlier).
  7. Result: Pod B throws Unexpected message id from the client. The client receives a generic error or hangs, forcing the user to manually refresh.

Expected Behavior

The application should gracefully recover from this desynchronization. If the UI state cannot be restored because the serialization never happened, the client should automatically trigger a UI State Re-synchronization (full page reload) rather than leaving the user in a broken state.

Technical Context

  • Environment: Kubernetes
  • Clustering: Hazelcast
  • Error: java.lang.IllegalStateException: Unexpected message id from the client

Activity

  1. mcollovati commented on Dec 15, 2025

    @mcollovati
    Contributor

    Reloading the page without notify the user seems not to be a good option for user experience.

    I wonder if this can be "fixed" in Flow, by throwing a specific exception when the issue happens, instead of illegal state exception. Then a new case can be added to system messages, so the developer can customize messages and behaviors (notification vs silent reload)

  2. moved this from 🪵Product backlog to 🟢Ready to Go in Vaadin Flow | Hilla | Kits ongoing workon Dec 18, 2025
  3. moved this from 🟢Ready to Go to 🪵Product backlog in Vaadin Flow | Hilla | Kits ongoing workon Jan 7, 2026
  4. added 2 commits that reference this issue on Jan 19, 2026
    8fcb1af
    5cf42a3
  5. moved this from 🪵Product backlog to ⚒️ In progress in Vaadin Flow | Hilla | Kits ongoing workon Jan 21, 2026
  6. moved this from ⚒️ In progress to 🔎Iteration reviews in Vaadin Flow | Hilla | Kits ongoing workon Jan 21, 2026
  7. moved this from Next for Dev. Team to Done / Pending Release in Vaadin Flow enhancements backlogon Jan 23, 2026
  8. added a commit that references this issue on Jan 23, 2026
    152f09d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions