Repository navigation
runtime: fix idle bookkeeping when a task dump wakes parked workers - #8372
Merged
Darksonn merged 4 commits intoSep 8, 2026
Merged
Conversation
Requesting a task dump calls `notify_all`, which unparks every worker thread directly rather than going through `Idle`. A worker that was parked at that point is woken to be traced while the scheduler still counts it as a sleeper. `transition_from_parked` did not account for that. With no tasks queued, the worker saw itself as still parked in `Idle` and returned without performing the logical unpark. Once tracing finished and the worker parked again, `transition_worker_to_parked` pushed it onto the sleeper list a second time and decremented `num_unparked` a second time, underflowing it. From then on `notify_should_wakeup` was always false, so no worker was ever notified of newly spawned work and the runtime silently stopped running tasks submitted from outside of it. Treat a worker woken for tracing like a worker woken with tasks queued and perform the logical unpark. This mirrors `transition_to_parked`, which already refuses to park a worker that is about to be traced. Co-authored-by: Isaac
`transition_to_parked` has refused to park a worker that is about to be traced for some time, but its comment did not say so. Mention it, and trim the comment added to `transition_from_parked` in the previous commit to match. Co-authored-by: Isaac
The bug fixed earlier in this PR was silent. `sleepers` is a `Vec`, so a worker can be tracked as a sleeper twice, and `dec_num_unparked` is an unchecked `fetch_sub`, so `num_unparked` wrapped rather than tripping on the second decrement. The runtime then believed every worker was busy and stopped notifying workers of new work, without any diagnostic. Assert both halves in debug builds: a worker is not already tracked as a sleeper when it parks, and `num_unparked` does not underflow. `Idle` already asserts the same class of invariant when it notifies a worker. Co-authored-by: Isaac
yzfeng2020
commented
Aug 19, 2026
|
|
||
| // A worker that is already tracked as a sleeper would be counted twice, | ||
| // which corrupts `num_unparked` and the sleeper list. | ||
| debug_assert!(!lock.idle.sleepers.contains(&worker)); |
Contributor
Author
There was a problem hiding this comment.
hey @conradludgate, i added two debug_assert! in addition to the fix. lmk what you think, thanks!
conradludgate
approved these changes
Aug 19, 2026
This was referenced Aug 19, 2026
Darksonn
enabled auto-merge (squash)
September 8, 2026 09:04
yzfeng2020
added a commit
to yzfeng2020/tokio
that referenced
this pull request
Sep 8, 2026
Resolve the conflict in tokio/tests/dump.rs: both sides appended a new regression test at the end of the file. Keep both `wedged_worker_during_tracing` from this branch and the `dump_of_parked_workers` module from tokio-rs#8372.
dadezzz
pushed a commit
to dadezzz/kube-dns-rs
that referenced
this pull request
Oct 20, 2026
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [tokio](https://tokio.rs) ([source](https://github.com/tokio-rs/tokio)) | dependencies | minor | `1.53.3` → `1.54.0` | --- ### Release Notes <details> <summary>tokio-rs/tokio (tokio)</summary> ### [`v1.54.0`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.54.0): Tokio v1.54.0 [Compare Source](tokio-rs/tokio@tokio-1.53.3...tokio-1.54.0) ### 1.54.0 (October 19th, 2026) The 1.54.0 release includes all changes up to 1.53.3. The MSRV is increased to 1.85. ([#​8514]) ##### Added - fs: add `File::set_times`, `File::set_modified`, and `fs::FileTimes` ([#​8504]) - net: implement `Default` for `named_pipe::{ClientOptions, ServerOptions}` ([#​8378]) - runtime: add `Builder::max_io_events_per_busy_tick` ([#​8410]) - runtime: stabilize `tokio::runtime::worker_index()` ([#​8431]) - sync: add `SemaphorePermit::semaphore()` ([#​8482]) - sync: add `blocking_acquire` methods to `Semaphore` ([#​8269]) - sync: add `watch::Receiver::is_closed` ([#​8481]) - wasm: support JSPI for Emscripten target ([#​8285], [#​8627]) - wasm: support the `wasm32-unknown-emscripten` target ([#​8281], [#​8422], [#​8528], [#​8628]) ##### Changed - fs: reject zero in `File::set_max_buf_size` ([#​8449]) - io: make `copy_buf` cooperative ([#​8459]) - io: panic when `BufReader` or `BufStream` is created with zero capacity ([#​8494]) - io: panic when in-memory pipe is created with zero capacity ([#​8397]) - io: reject zero buffer size in `copy_bidirectional_with_sizes` ([#​8454]) - io: support vectored writes in `Sink` ([#​8663]) - net: start `UnixStream::pair` and `UnixDatagram::pair` sockets as writable ([#​8444]) - net: start accepted sockets as readable and writable ([#​8435]) - net: support `UCred` on Fuchsia ([#​8257]) - process: use `mio::unix::pipe` for Unix child stdio ([#​8603]) - runtime: cache-pad the shared injection queue and idle state ([#​8622]) - runtime: decide auto-boxing of spawned futures at monomorphization time ([#​8414]) - runtime: make `enable_io` available whenever `rt` is enabled ([#​8387]) - runtime: move `block_in_place` setup into a non-generic function ([#​8315]) - runtime: move the multi-thread inject queue to its own mutex ([#​8382]) - runtime: panic when `max_io_events_per_tick` is zero ([#​8476]) - runtime: suggest `.await` in nested `block_on` panic message ([#​8501]) - sync: avoid atomic reader count in broadcast slots ([#​8357]) - sync: use acquire/release orderings in `Notify` ([#​8325]) - time: remove the runtime handle from timers ([#​8458]) ##### Fixed - fs: restore `File` internal state when op fails ([#​8291]) - io: complete zero-length memory stream operations immediately ([#​8323]) - io: flush the writer when the reader stalls in `copy_buf` ([#​8424]) - io: keep the shutdown bit when clearing readiness ([#​8435]) - io: preserve the full `ScheduledIo` tick in readiness events ([#​8331]) - io: read lines as bytes so a partial line survives an I/O error ([#​8400]) - io: retry on `ErrorKind::Interrupted` in more I/O operations ([#​8417], [#​8432], [#​8448], [#​8453], [#​8455], [#​8460]) - io: use `spawn_mandatory_blocking` for stdout and stderr ([#​8506]) - io: wait for pending operations before `BufReader` seeks ([#​8599]) - net: close `CreateFileW` handle on `SetNamedPipeHandleState` failure ([#​8418]) - net: fix peer credentials on NetBSD ([#​8589]) - net: saturate `TcpSocket` and `UnixSocket` listen backlog ([#​8511]) - net: treat `ENOTCONN` as success in `UnixStream::shutdown` ([#​8526]) - runtime: allow building with `--all-features` without `tokio_unstable` ([#​8401]) - runtime: avoid overflow in `shutdown_timeout()` ([#​8497]) - runtime: fix `spawn_blocking` hang when only scheduler workers exist ([#​8408]) - runtime: fix misleading signal driver panic message ([#​8419]) - sync: preserve FIFO lock order when coop budget is exhausted ([#​8371]) - sync: use `usize` for permit count in `SemaphorePermit` ([#​8405]) - time: avoid interval deadline overflow ([#​8385]) - time: handle wrapped top-level timer-wheel slots ([#​8334]) ##### Unstable - fs: fall back when `io_uring` has no IO driver ([#​8165]) - metrics: cap log histogram precision ([#​8496]) - runtime: add an opt-in sharded `spawn_blocking` queue ([#​8337]) - runtime: expose schedule latency in task hooks ([#​8282]) - taskdump: fix idle bookkeeping when a task dump wakes parked workers ([#​8372]) - taskdump: skip non-idle tasks during taskdumps ([#​8568]) ##### Documented - docs: fix stray braces and broken links in rustdoc ([#​8605]) - fs: document cancellation behavior of `tokio::fs` ([#​8265]) - net: add Safety docs for named pipe `from_raw_handle` ([#​8378]) - net: document that `TcpStream::connect` returns only the last error ([#​8421]) - runtime: add error documentation for `runtime::Builder` ([#​8297]) - runtime: link `available_parallelism` in worker count docs ([#​8625]) - sync: clarify panic behavior of `broadcast::channel()` ([#​8420]) - sync: fix typo in `Semaphore` documentation ([#​8521]) [#​8165]: tokio-rs/tokio#8165 [#​8257]: tokio-rs/tokio#8257 [#​8265]: tokio-rs/tokio#8265 [#​8269]: tokio-rs/tokio#8269 [#​8281]: tokio-rs/tokio#8281 [#​8282]: tokio-rs/tokio#8282 [#​8285]: tokio-rs/tokio#8285 [#​8291]: tokio-rs/tokio#8291 [#​8297]: tokio-rs/tokio#8297 [#​8315]: tokio-rs/tokio#8315 [#​8323]: tokio-rs/tokio#8323 [#​8325]: tokio-rs/tokio#8325 [#​8331]: tokio-rs/tokio#8331 [#​8334]: tokio-rs/tokio#8334 [#​8337]: tokio-rs/tokio#8337 [#​8357]: tokio-rs/tokio#8357 [#​8371]: tokio-rs/tokio#8371 [#​8372]: tokio-rs/tokio#8372 [#​8378]: tokio-rs/tokio#8378 [#​8382]: tokio-rs/tokio#8382 [#​8385]: tokio-rs/tokio#8385 [#​8387]: tokio-rs/tokio#8387 [#​8397]: tokio-rs/tokio#8397 [#​8400]: tokio-rs/tokio#8400 [#​8401]: tokio-rs/tokio#8401 [#​8405]: tokio-rs/tokio#8405 [#​8408]: tokio-rs/tokio#8408 [#​8410]: tokio-rs/tokio#8410 [#​8414]: tokio-rs/tokio#8414 [#​8417]: tokio-rs/tokio#8417 [#​8418]: tokio-rs/tokio#8418 [#​8419]: tokio-rs/tokio#8419 [#​8420]: tokio-rs/tokio#8420 [#​8421]: tokio-rs/tokio#8421 [#​8422]: tokio-rs/tokio#8422 [#​8424]: tokio-rs/tokio#8424 [#​8431]: tokio-rs/tokio#8431 [#​8432]: tokio-rs/tokio#8432 [#​8435]: tokio-rs/tokio#8435 [#​8444]: tokio-rs/tokio#8444 [#​8448]: tokio-rs/tokio#8448 [#​8449]: tokio-rs/tokio#8449 [#​8453]: tokio-rs/tokio#8453 [#​8454]: tokio-rs/tokio#8454 [#​8455]: tokio-rs/tokio#8455 [#​8458]: tokio-rs/tokio#8458 [#​8459]: tokio-rs/tokio#8459 [#​8460]: tokio-rs/tokio#8460 [#​8476]: tokio-rs/tokio#8476 [#​8481]: tokio-rs/tokio#8481 [#​8482]: tokio-rs/tokio#8482 [#​8494]: tokio-rs/tokio#8494 [#​8496]: tokio-rs/tokio#8496 [#​8497]: tokio-rs/tokio#8497 [#​8501]: tokio-rs/tokio#8501 [#​8504]: tokio-rs/tokio#8504 [#​8506]: tokio-rs/tokio#8506 [#​8511]: tokio-rs/tokio#8511 [#​8514]: tokio-rs/tokio#8514 [#​8521]: tokio-rs/tokio#8521 [#​8526]: tokio-rs/tokio#8526 [#​8528]: tokio-rs/tokio#8528 [#​8568]: tokio-rs/tokio#8568 [#​8589]: tokio-rs/tokio#8589 [#​8599]: tokio-rs/tokio#8599 [#​8603]: tokio-rs/tokio#8603 [#​8605]: tokio-rs/tokio#8605 [#​8622]: tokio-rs/tokio#8622 [#​8625]: tokio-rs/tokio#8625 [#​8627]: tokio-rs/tokio#8627 [#​8628]: tokio-rs/tokio#8628 [#​8663]: tokio-rs/tokio#8663 </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate CLI](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0NC4xNjcuNSIsInVwZGF0ZWRJblZlciI6IjQ0LjE2Ny41IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6W119-->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
On the multi-threaded runtime, a task dump taken while a worker is idle corrupts the scheduler's idle bookkeeping, after which tasks spawned from outside the runtime never run.
Handle::dumpcallsnotify_all, which unparks worker threads directly instead of going throughIdle, so a parked worker is woken for tracing while the scheduler still counts it as a sleeper.transition_from_parkeddoesn't accountfor that: with no tasks queued it returns without performing the logical unpark, so when the worker parks again after tracing it is pushed onto
sleeperstwice andnum_unparkedis decremented twice, underflowing. From then onnotify_should_wakeupis always false and no worker is ever notified of new work.Solution
Treat a worker woken for tracing like a worker woken with tasks queued and perform the logical unpark. This mirrors
transition_to_parked, which already refuses to park a worker that is about to be traced.The regression tests in
tokio/tests/dump.rspark every worker, take a dump, wait for the workers to be traced and re-park, then assert that a task spawned from outside the runtime still runs.The
current_threadscheduler is unaffected; it has no idle bookkeeping of this kind.