Repository navigation
runtime: optional eager I/O driver/timer handoff when polling tasks - #8010
Conversation
|
Oh I really do not love that my reproducer apparently cannot make the runtime deadlock on ARM --- that suggests some difference of behavior there which is also disconcerting. |
This way, it should be less susceptible to some platforms having slow enough timers that the two tasks are woken in the same iteration of the timer wheel, or whatever it is that broke CI.
I suspect this was due to the test's previous reliance on timers; if the test was running in a sufficiently slow environment (such as a QEMU VM on a GitHub Acions runner), the two timers could conceivably complete in the same poll of the timer wheel, resulting in it "not deadlocking". I've rewritten the test to use Unix pipes instead which should hopefully be less flaky. |
Darksonn
left a comment
There was a problem hiding this comment.
Implementation looks pretty clean to me, thanks!
I'll leave this for a few days in case anyone else wants to review it, but otherwise I'll merge this soon.
|
FWIW as a random user, thank you! I ran into the problem this fixes when testing a transaction system. My test got stuck with one task hogging one thread, even though many idle threads were available. I knew I had an unyielding retry loop (which I should fix), but I continued to dig in to explain how one task could wedge the entire runtime. The scenario was a single task in an unyielding retry loop, but a timer was set that should have unblocked another task that would get this one unstuck. At least in my case, this PR solves the problem:
I think the PR makes sense conceptually and should make the runtime behavior more predictable for users. I can't speak to how it affects normal performance, however. I hope this can make its way to becoming easily enabled (without |
I would certainly like to expose this behavior without the unstable flag, and possibly as a default eventually: personally, I also feel that this behavior is probably correct for a majority of applications, and I expect it will probably result in better overall performance in most cases. However, I wanted to be a bit conservative when introducing a change to the runtime's behavior --- even if we confirm this to be an improvement in most cases, we may want to change the interface for enabling it, for example. I'm also hoping there's a path to dropping the unstable flag after it's been out for a couple releases. |
|
I think if people try it out and confirm no serious perf regressions, we can just make this the default. It won't break any workflows. It's a bit awkward taking into account that io-uring wants us to introduce per-thread IO contexts, which would bring back #6315 unless we make sure that threads have a way to check the ring owned by other threads for completions to avoid starvation. But probably this means that per-thread rings must be guarded behind a new runtime flavor and the default Tokio configuration will just have a single shared ring. |
That sounds great, we could always end up changing the builder option to be |
# 1.52.0 (April 14th, 2026) ## Added - io: `AioSource::register_borrowed` for I/O safety support ([#7992]) - net: add `try_io` function to `unix::pipe` sender and receiver types ([#8030]) ## Added (unstable) - runtime: `Builder::enable_eager_driver_handoff` setting enable eager hand off of the I/O and time drivers before polling tasks ([#8010]) - taskdump: add `trace_with()` for customized task dumps ([#8025]) - taskdump: allow `impl FnMut()` in `trace_with`` instead of just `fn()` ([#8040]) - fs: support `io_uring` in `AsyncRead` for `File` ([#7907]) ## Changed - runtime: improve `spawn_blocking` scalability with sharded queue ([#7757]) - runtime: use `compare_exchange_weak()` in worker queue ([#8028]) ## Fixed - runtime: overflow second half of tasks when local queue is filled instead of first half ([#8029]) ## Documented - docs: fix typo in `oneshot::Sender::send` docs ([#8026]) - docs: hide #[tokio::main] attribute in the docs of `sync::watch` ([#8035]) - net: add docs on `ConnectionRefused` errors with UDP sockets ([#7870]) [#7757]: #7757 [#7870]: #7870 [#7907]: #7907 [#7992]: #7992 [#8010]: #8010 [#8025]: #8025 [#8026]: #8026 [#8028]: #8028 [#8029]: #8029 [#8030]: #8030 [#8035]: #8035 [#8040]: #8040
# 1.52.0 (April 14th, 2026) ## Added - io: `AioSource::register_borrowed` for I/O safety support ([#7992]) - net: add `try_io` function to `unix::pipe` sender and receiver types ([#8030]) ## Added (unstable) - runtime: `Builder::enable_eager_driver_handoff` setting enable eager hand off of the I/O and time drivers before polling tasks ([#8010]) - taskdump: add `trace_with()` for customized task dumps ([#8025]) - taskdump: allow `impl FnMut()` in `trace_with` instead of just `fn()` ([#8040]) - fs: support `io_uring` in `AsyncRead` for `File` ([#7907]) ## Changed - runtime: improve `spawn_blocking` scalability with sharded queue ([#7757]) - runtime: use `compare_exchange_weak()` in worker queue ([#8028]) ## Fixed - runtime: overflow second half of tasks when local queue is filled instead of first half ([#8029]) ## Documented - docs: fix typo in `oneshot::Sender::send` docs ([#8026]) - docs: hide #[tokio::main] attribute in the docs of `sync::watch` ([#8035]) - net: add docs on `ConnectionRefused` errors with UDP sockets ([#7870]) [#7757]: #7757 [#7870]: #7870 [#7907]: #7907 [#7992]: #7992 [#8010]: #8010 [#8025]: #8025 [#8026]: #8026 [#8028]: #8028 [#8029]: #8029 [#8030]: #8030 [#8035]: #8035 [#8040]: #8040
This updates our minimum Tokio version to [1.52.0]. This allows us to pick up two major fixes that will change our default runtime configuration: - tokio-rs/tokio#8010 (released in [1.52.0]) which fixes oxidecomputer/omicron#9619 when its builder option is enabled, - tokio-rs/tokio#7431,(released in 1.51.0), which allows tasks in the LIFO slot to participate in work-stealing. Subsequent commits will actually update our runtime settings after picking up these releases. [1.52.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0 [1.51.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0
This updates our minimum Tokio version to [1.52.0]. This allows us to pick up two major fixes that will change our default runtime configuration: - tokio-rs/tokio#8010 (released in [1.52.0]) which fixes oxidecomputer/omicron#9619 when its builder option is enabled, - tokio-rs/tokio#7431,(released in 1.51.0), which allows tasks in the LIFO slot to participate in work-stealing. Subsequent commits will actually update our runtime settings after picking up these releases. [1.52.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0 [1.51.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0
This commit enables the eager I/O and time driver hand-off setting for the multi-threaded Tokio runtime, which I added in tokio-rs/tokio#8010. This is currently an experimental feature, although it is your author's opinion that this is really a fix for incorrect runtime behavior. It changes worker threads in the multi-threaded runtime to wake another worker prior to polling tasks if that worker had previously been parked on the I/O driver or timer wheel. Eagerly handing off these resources should prevent pathologies such as omicron#9619.
This updates our minimum Tokio version to [1.52.0]. This allows us to pick up two major fixes that will change our default runtime configuration: - tokio-rs/tokio#8010 (released in [1.52.0]) which fixes oxidecomputer/omicron#9619 when its builder option is enabled, - tokio-rs/tokio#7431,(released in 1.51.0), which allows tasks in the LIFO slot to participate in work-stealing. Subsequent commits will actually update our runtime settings after picking up these releases. [1.52.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0 [1.51.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0
This commit enables the eager I/O and time driver hand-off setting for the multi-threaded Tokio runtime, which I added in tokio-rs/tokio#8010. This is currently an experimental feature, although it is your author's opinion that this is really a fix for incorrect runtime behavior. It changes worker threads in the multi-threaded runtime to wake another worker prior to polling tasks if that worker had previously been parked on the I/O driver or timer wheel. Eagerly handing off these resources should prevent pathologies such as omicron#9619.
update `oxide-tokio-rt` to v0.1.4, `tokio` to v1.52.0 This branch updates our dependency on `oxide-tokio-rt` to pick up Tokio v1.52.0 and the corresponding changes to the default runtime settings in `oxide-tokio-rt`. In particular, this allows us to pick up two of my upstream fixes in Tokio for a pair of issues that have been major thorns in our side for some time: * Tokio PR tokio-rs/tokio#7431, released in [Tokio v1.51.0], changes the multi-threaded runtime to allow tasks in the LIFO slot to participate in work-stealing. Therefore, it should no longer be necessary to disable the LIFO slot optimization, as the pathology described in #8334, where a task placed in the LIFO slot can become permanently or semi-permanently stuck while the task that notified them runs for a long time without yielding, can no longer occur. `oxide-tokio-rt` v0.1.4 removes the runtime configuration to disable the LIFO slot as the issue has been fixed upstream. * Tokio PR tokio-rs/tokio#8010, released in [Tokio v1.52.0], which adds eager handoff for the I/O and time drivers in the multi-threaded runtime. This is currently an experimental feature, although it is your author's opinion that this is really a fix for incorrect runtime behavior. It changes worker threads in the multi-threaded runtime to wake another worker prior to polling tasks if that worker had previously been parked on the I/O driver or timer wheel. Eagerly handing off these resources should prevent pathologies such as #9619. `oxide-tokio-rt` v0.1.4 enables this behavior by default. Fixes #8334 Fixes #9619 [Tokio v1.52.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0 [Tokio v1.51.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0
This branch updates our dependency on `oxide-tokio-rt` to pick up Tokio v1.52.0 and the corresponding changes to the default runtime settings in `oxide-tokio-rt`. In particular, this allows us to pick up two of my upstream fixes in Tokio for a pair of issues that have been major thorns in our side for some time: * Tokio PR tokio-rs/tokio#7431, released in [Tokio v1.51.0], changes the multi-threaded runtime to allow tasks in the LIFO slot to participate in work-stealing. Therefore, it should no longer be necessary to disable the LIFO slot optimization, as the pathology described in #8334, where a task placed in the LIFO slot can become permanently or semi-permanently stuck while the task that notified them runs for a long time without yielding, can no longer occur. `oxide-tokio-rt` v0.1.4 removes the runtime configuration to disable the LIFO slot as the issue has been fixed upstream. * Tokio PR tokio-rs/tokio#8010, released in [Tokio v1.52.0], which adds eager handoff for the I/O and time drivers in the multi-threaded runtime. This is currently an experimental feature, although it is your author's opinion that this is really a fix for incorrect runtime behavior.[^1] It changes worker threads in the multi-threaded runtime to wake another worker prior to polling tasks if that worker had previously been parked on the I/O driver or timer wheel. Eagerly handing off these resources should prevent pathologies such as #9619. `oxide-tokio-rt` v0.1.4 enables this behavior by default. Fixes #8334 Fixes #9619 [Tokio v1.52.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0 [Tokio v1.51.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0 [^1]: Cue @leftwo's dictum that a "performance regression" from fixing incorrect behavior...isn't a performance regression.
Commit #10272 updated our dependency on `oxide-tokio-rt` to v0.1.4 and our `tokio` dependency to v1.52.0. This allowed us to pick up two of my fixes for Tokio issues that have been a thorn in our side for a long time, tokio-rs/tokio#7431 and tokio-rs/tokio#8010, which fix #8334 and #9619, respectively. The nature of these fixes is described in greater detail in #10272. Unfortunately, #10272 had to be reverted (in #10279), since @iliana discovered an unrelated regression in Tokio v1.52.0, tokio-rs/tokio#8056 (our issue #10277). This regression caused `spawn_blocking` to occasionally hang, and was introduced in tokio-rs/tokio@1604bc3 (PR tokio-rs/tokio#7757). I've since reverted this change upstream (tokio-rs/tokio#8057), and published a patch release ([v1.52.1]), which fixes the regression. Therefore, it is now once again safe to update our Tokio dependency to pick up the other fixes. This commit does that. I've also confirmed that the issue described in #10277 is no longer present in Tokio v1.52.1, as demonstrated by the fact `cargo nextest run -p omicron-sled-agent --stress-count 100 -- --exact artifact_store::test::issue_7796` now succeeds without hanging once again. Fixes #10272
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [tokio](https://tokio.rs) ([source](https://github.com/tokio-rs/tokio)) | workspace.dependencies | minor | `1.51.1` → `1.52.1` | --- ### Release Notes <details> <summary>tokio-rs/tokio (tokio)</summary> ### [`v1.52.1`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.1): Tokio v1.52.1 [Compare Source](tokio-rs/tokio@tokio-1.52.0...tokio-1.52.1) ### 1.52.1 (April 16th, 2026) #### Fixed - runtime: revert [#​7757] to fix [a regression][#​8056] that causes `spawn_blocking` to hang ([#​8057]) [#​7757]: tokio-rs/tokio#7757 [#​8056]: tokio-rs/tokio#8056 [#​8057]: tokio-rs/tokio#8057 ### [`v1.52.0`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0): Tokio v1.52.0 [Compare Source](tokio-rs/tokio@tokio-1.51.1...tokio-1.52.0) ### 1.52.0 (April 14th, 2026) #### Added - io: `AioSource::register_borrowed` for I/O safety support ([#​7992]) - net: add `try_io` function to `unix::pipe` sender and receiver types ([#​8030]) #### Added (unstable) - runtime: `Builder::enable_eager_driver_handoff` setting enable eager hand off of the I/O and time drivers before polling tasks ([#​8010]) - taskdump: add `trace_with()` for customized task dumps ([#​8025]) - taskdump: allow `impl FnMut()` in `trace_with` instead of just `fn()` ([#​8040]) - fs: support `io_uring` in `AsyncRead` for `File` ([#​7907]) #### Changed - runtime: improve `spawn_blocking` scalability with sharded queue ([#​7757]) - runtime: use `compare_exchange_weak()` in worker queue ([#​8028]) #### Fixed - runtime: overflow second half of tasks when local queue is filled instead of first half ([#​8029]) #### Documented - docs: fix typo in `oneshot::Sender::send` docs ([#​8026]) - docs: hide #\[tokio::main] attribute in the docs of `sync::watch` ([#​8035]) - net: add docs on `ConnectionRefused` errors with UDP sockets ([#​7870]) [#​7757]: tokio-rs/tokio#7757 [#​7870]: tokio-rs/tokio#7870 [#​7907]: tokio-rs/tokio#7907 [#​7992]: tokio-rs/tokio#7992 [#​8010]: tokio-rs/tokio#8010 [#​8025]: tokio-rs/tokio#8025 [#​8026]: tokio-rs/tokio#8026 [#​8028]: tokio-rs/tokio#8028 [#​8029]: tokio-rs/tokio#8029 [#​8030]: tokio-rs/tokio#8030 [#​8035]: tokio-rs/tokio#8035 [#​8040]: tokio-rs/tokio#8040 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xMDEuMSIsInVwZGF0ZWRJblZlciI6IjQzLjEwMS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJ0eXBlL21pbm9yIl19--> Reviewed-on: https://git.erwanleboucher.dev/eleboucher/towonel/pulls/11
In order to re-enable stealing tasks from the LIFO slot, we would like to allow the user to configure whether or not pushing tasks to the LIFO slot unparks another worker thread. This is in order to avoid performance regressions due to the overhead of additional cross-thread wakeups (i.e. #8065). In [this comment][1] on #8092, @carllerche suggested that we do this by having a single option controlling whether we wake parked workers more or less aggressively which would also control the eager I/O and time driver handoff behavior from #8010, which also causes the runtime to wake parked workers more aggressively in order to prevent potential deadlocks and task starvation. This PR adds such a knob to the `runtime::Builder` type, which requires `tokio_unstable`. This is represented as an `UnparkingMode` enum. We use an enum rather than a `bool` primarily because I would like to add a third unparking mode in the future. This mode would wake parked workers much less frequently, but would instead track whether there are any stealable tasks enqueued, and wake a stealer only when _transitioning_ from 0-1 stealable tasks, and have parking workers park with a timeout while we are in the "stealable tasks" state. The idea is that we would prevent deadlocks while avoiding spurious IPIs in this case. But, I haven't actually implemented this behavior yet, so presently, the enum has two variants: `Traditional`, which is the current runtime behavior, and `Cautious`, which wakes more aggressively to try to prevent deadlocks (and enables waking on LIFO pushes and on I/O driver handoff). The existing `enable_eager_driver_handoff` is deprecated in favor of the new API. I also felt that the enum made it a bit easier to document the different behaviors. If anyone has input on how we can better explain the two unparking modes to users, I'd welcome suggestions! Closes #8118 [1]: #8092 (review)
In order to re-enable stealing tasks from the LIFO slot, we would like to allow the user to configure whether or not pushing tasks to the LIFO slot unparks another worker thread. This is in order to avoid performance regressions due to the overhead of additional cross-thread wakeups (i.e. #8065). In [this comment][1] on #8092, @carllerche suggested that we do this by having a single option controlling whether we wake parked workers more or less aggressively which would also control the eager I/O and time driver handoff behavior from #8010, which also causes the runtime to wake parked workers more aggressively in order to prevent potential deadlocks and task starvation. This PR adds such a knob to the `runtime::Builder` type, which requires `tokio_unstable`. This is represented as an `UnparkingMode` enum. We use an enum rather than a `bool` primarily because I would like to add a third unparking mode in the future. This mode would wake parked workers much less frequently, but would instead track whether there are any stealable tasks enqueued, and wake a stealer only when _transitioning_ from 0-1 stealable tasks, and have parking workers park with a timeout while we are in the "stealable tasks" state. The idea is that we would prevent deadlocks while avoiding spurious IPIs in this case. But, I haven't actually implemented this behavior yet, so presently, the enum has two variants: `Traditional`, which is the current runtime behavior, and `Cautious`, which wakes more aggressively to try to prevent deadlocks (and enables waking on LIFO pushes and on I/O driver handoff). The existing `enable_eager_driver_handoff` is deprecated in favor of the new API. I also felt that the enum made it a bit easier to document the different behaviors. If anyone has input on how we can better explain the two unparking modes to users, I'd welcome suggestions! Closes #8118 [1]: #8092 (review)
[cb skip] Changes in 1.53.1 ----------------- 1.53.1 (July 20th, 2026) ------------------------ Fixed ----- - signal: restore MSRV by removing `OnceLock::wait` from the Windows handler ([#8300]) Fixed (unstable) ---------------- - time: fix alt timer cancellation and insertion race ([#8252]) Documented ---------- - runtime: remove dead link definition in Runtime::block_on ([#8301]) [#8252]: tokio-rs/tokio#8252 [#8300]: tokio-rs/tokio#8300 [#8301]: tokio-rs/tokio#8301 Changes in 1.53.0 ----------------- 1.53.0 (July 17th, 2026) ------------------------ Added ----- - fs: implement `From<OwnedFd>` and `From<OwnedHandle>` for `File` ([#8266]) - metrics: add task schedule latency metric ([#7986]) - net: add `SocketAddr` methods to Unix sockets ([#8144]) Changed ------- - io: add `#[inline]` to IO trait impls for in-memory types ([#8242]) - net: implement UCred::pid on FreeBSD ([#8086]) - net: support Nuttx target os ([#8259]) - signal: refactor global variables on Windows ([#8231]) - sync: `mpsc::{Receiver,UnboundedReceiver}` now drops waker on drop, even if there are still senders ([#8095]) - taskdump: support taskdumps on s390x ([#8192]) - time: add `#[track_caller]` to `timeout_at()` ([#8077]) - time: consolidate mutex locks on spurious poll ([#8124]) - time: defer waker clone on spurious poll ([#8107]) - time: move lazy-registration state into `Sleep` ([#8132]) - tracing: remove unnecessary span clone ([#8126]) Fixed ----- - io: do not treat zero-length reads as EOF in `Chain` ([#8251]) - net: use getpeereid for QNX peer credentials ([#8270]) - runtime: avoid illegal state in `FastRand` ([#8078]) - sync: wake mpsc receiver when a queued `reserve[_many]` returns permits ([#8260]) - taskdump: skip double wake on `Trace::capture`/`Trace::trace_with` ([#8043]) - time: avoid stack overflow in runtime constructor ([#8093]) - time (alt timer): ensure timers stay in the same runtime after `.reset()` ([#8169]) IO uring (unstable) ------------------- - fs: use io-uring for `fs::try_exists` ([#8080]) - fs: use io-uring for renaming files ([#7800]) - rt: flush io-uring CQE in case of CQE overflow ([#8277]) Documented ---------- - docs: clarify cancel safety wording ([#8181]) - fs: clarify `create_dir_all` succeeds if path exists ([#8149]) - io: add warning about stdout reordering with multiple handles ([#8276]) - net: document pipe `try_read*`/`try_write*` readiness behavior ([#8032]) - runtime: document interaction with fork() ([#8202]) - sync: clarify broadcast lagging semantics ([#8239]) - sync: document memory ordering guarantees for Semaphore ([#8119]) - task: explain why `yield_now` defers its waker ([#8254]) - time: add panic docs to `timeout_at()` ([#8077]) - time: fix reversed poll order in timeout doc ([#8214]) [#7800]: tokio-rs/tokio#7800 [#7986]: tokio-rs/tokio#7986 [#8032]: tokio-rs/tokio#8032 [#8043]: tokio-rs/tokio#8043 [#8077]: tokio-rs/tokio#8077 [#8078]: tokio-rs/tokio#8078 [#8080]: tokio-rs/tokio#8080 [#8086]: tokio-rs/tokio#8086 [#8093]: tokio-rs/tokio#8093 [#8095]: tokio-rs/tokio#8095 [#8107]: tokio-rs/tokio#8107 [#8119]: tokio-rs/tokio#8119 [#8124]: tokio-rs/tokio#8124 [#8126]: tokio-rs/tokio#8126 [#8132]: tokio-rs/tokio#8132 [#8144]: tokio-rs/tokio#8144 [#8149]: tokio-rs/tokio#8149 [#8169]: tokio-rs/tokio#8169 [#8181]: tokio-rs/tokio#8181 [#8192]: tokio-rs/tokio#8192 [#8193]: tokio-rs/tokio#8193 [#8202]: tokio-rs/tokio#8202 [#8214]: tokio-rs/tokio#8214 [#8231]: tokio-rs/tokio#8231 [#8239]: tokio-rs/tokio#8239 [#8242]: tokio-rs/tokio#8242 [#8251]: tokio-rs/tokio#8251 [#8254]: tokio-rs/tokio#8254 [#8259]: tokio-rs/tokio#8259 [#8260]: tokio-rs/tokio#8260 [#8266]: tokio-rs/tokio#8266 [#8270]: tokio-rs/tokio#8270 [#8276]: tokio-rs/tokio#8276 [#8277]: tokio-rs/tokio#8277 Changes in 1.52.4 ----------------- 1.52.4 (July 16th, 2026) ------------------------ Fixed ----- - runtime: don't skip the driver when `before_park` schedules work ([#8222]) Fixed (unstable) ---------------- - taskdump: remove crate disambiguators from output ([#8264]) [#8264]: tokio-rs/tokio#8264 [#8222]: tokio-rs/tokio#8222 Changes in 1.52.3 ----------------- 1.52.3 (May 8th, 2026) ---------------------- Fixed ----- * sync: fix underflow in mpsc channel `len()` ([#8062]) * sync: notify receivers in mpsc `OwnedPermit::release()` method ([#8075]) * sync: require that an `RwLock` has `max_readers != 0` ([#8076]) * sync: return `Empty` from `try_recv()` when mpsc is closed with outstanding permits ([#8074]) [#8062]: tokio-rs/tokio#8062 [#8074]: tokio-rs/tokio#8074 [#8075]: tokio-rs/tokio#8075 [#8076]: tokio-rs/tokio#8076 Changes in 1.52.2 ----------------- 1.52.2 (May 4th, 2026) ---------------------- This release reverts the LIFO slot stealing change introduced in 1.51.0 ([#7431]), due to [its performance impact][#8065]. ([#8100]) [#7431]: tokio-rs/tokio#7431 [#8065]: tokio-rs/tokio#8065 [#8100]: tokio-rs/tokio#8100 Changes in 1.52.1 ----------------- 1.52.1 (April 16th, 2026) ------------------------- Fixed ----- - runtime: revert [#7757] to fix [a regression][#8056] that causes `spawn_blocking` to hang ([#8057]) [#7757]: tokio-rs/tokio#7757 [#8056]: tokio-rs/tokio#8056 [#8057]: tokio-rs/tokio#8057 Changes in 1.52.0 ----------------- 1.52.0 (April 14th, 2026) ------------------------- Added ----- - io: `AioSource::register_borrowed` for I/O safety support ([#7992]) - net: add `try_io` function to `unix::pipe` sender and receiver types ([#8030]) Added (unstable) ---------------- - runtime: `Builder::enable_eager_driver_handoff` setting enable eager hand off of the I/O and time drivers before polling tasks ([#8010]) - taskdump: add `trace_with()` for customized task dumps ([#8025]) - taskdump: allow `impl FnMut()` in `trace_with` instead of just `fn()` ([#8040]) - fs: support `io_uring` in `AsyncRead` for `File` ([#7907]) Changed ------- - runtime: improve `spawn_blocking` scalability with sharded queue ([#7757]) - runtime: use `compare_exchange_weak()` in worker queue ([#8028]) Fixed ----- - runtime: overflow second half of tasks when local queue is filled instead of first half ([#8029]) Documented ---------- - docs: fix typo in `oneshot::Sender::send` docs ([#8026]) - docs: hide #[tokio::main] attribute in the docs of `sync::watch` ([#8035]) - net: add docs on `ConnectionRefused` errors with UDP sockets ([#7870]) [#7757]: tokio-rs/tokio#7757 [#7870]: tokio-rs/tokio#7870 [#7907]: tokio-rs/tokio#7907 [#7992]: tokio-rs/tokio#7992 [#8010]: tokio-rs/tokio#8010 [#8025]: tokio-rs/tokio#8025 [#8026]: tokio-rs/tokio#8026 [#8028]: tokio-rs/tokio#8028 [#8029]: tokio-rs/tokio#8029 [#8030]: tokio-rs/tokio#8030 [#8035]: tokio-rs/tokio#8035 [#8040]: tokio-rs/tokio#8040 Changes in 1.51.4 ----------------- 1.51.4 (July 16th, 2026) ------------------------ Fixed ----- - runtime: don't skip the driver when `before_park` schedules work ([#8222]) [#8222]: tokio-rs/tokio#8222 Changes in 1.51.3 ----------------- 1.51.3 (May 8th, 2026) ---------------------- Fixed ----- * sync: fix underflow in mpsc channel `len()` ([#8062]) * sync: notify receivers in mpsc `OwnedPermit::release()` method ([#8075]) * sync: require that an `RwLock` has `max_readers != 0` ([#8076]) * sync: return `Empty` from `try_recv()` when mpsc is closed with outstanding permits ([#8074]) [#8062]: tokio-rs/tokio#8062 [#8074]: tokio-rs/tokio#8074 [#8075]: tokio-rs/tokio#8075 [#8076]: tokio-rs/tokio#8076 Changes in 1.51.2 ----------------- 1.51.2 (May 4th, 2026) ---------------------- This release reverts the LIFO slot stealing change introduced in 1.51.0 ([#7431]), due to [its performance impact][#8065]. ([#8100]) [#7431]: tokio-rs/tokio#7431 [#8065]: tokio-rs/tokio#8065 [#8100]: tokio-rs/tokio#8100 Changes in 1.51.1 ----------------- 1.51.1 (April 8th, 2026) ------------------------ Fixed ----- - sync: fix semaphore reopens after forget ([#8021]) - net: surface errors from `SO_ERROR` on `recv` for UDP sockets on Linux ([#8001]) Fixed (unstable) ---------------- - metrics: fix `worker_local_schedule_count` test ([#8008]) - rt: do not leak fd when cancelling io\_uring open operation ([#7983]) [#7983]: tokio-rs/tokio#7983 [#8001]: tokio-rs/tokio#8001 [#8008]: tokio-rs/tokio#8008 [#8021]: tokio-rs/tokio#8021 Changes in 1.51.0 ----------------- 1.51.0 (April 3rd, 2026) ------------------------ Added ----- - net: implement `get_peer_cred` on Hurd ([#7989]) - runtime: add `tokio::runtime::worker_index()` ([#7921]) - runtime: add runtime name ([#7924]) - runtime: stabilize `LocalRuntime` ([#7557]) - wasm: add wasm32-wasip2 networking support ([#7933]) Changed ------- - runtime: steal tasks from the LIFO slot ([#7431]) Fixed ----- - docs: do not show "Available on non-loom only." doc label ([#7977]) - macros: improve overall macro hygiene ([#7997]) - sync: fix `notify_waiters` priority in `Notify` ([#7996]) - sync: fix panic in `Chan::recv_many` when called with non-empty vector on closed channel ([#7991]) [#7431]: tokio-rs/tokio#7431 [#7557]: tokio-rs/tokio#7557 [#7921]: tokio-rs/tokio#7921 [#7924]: tokio-rs/tokio#7924 [#7933]: tokio-rs/tokio#7933 [#7977]: tokio-rs/tokio#7977 [#7989]: tokio-rs/tokio#7989 [#7991]: tokio-rs/tokio#7991 [#7996]: tokio-rs/tokio#7996 [#7997]: tokio-rs/tokio#7997 Changes in 1.50.0 ----------------- 1.50.0 (Mar 3rd, 2026) ---------------------- Added ----- - net: add `TcpStream::set_zero_linger` ([#7837]) - rt: add `is_rt_shutdown_err` ([#7771]) Changed ------- - io: add optimizer hint that `memchr` returns in-bounds pointer ([#7792]) - io: implement vectored writes for `write_buf` ([#7871]) - runtime: panic when `event_interval` is set to 0 ([#7838]) - runtime: shorten default thread name to fit in Linux limit ([#7880]) - signal: remember the result of `SetConsoleCtrlHandler` ([#7833]) - signal: specialize windows `Registry` ([#7885]) Fixed ----- - io: always cleanup `AsyncFd` registration list on deregister ([#7773]) - macros: remove (most) local `use` declarations in `tokio::select!` ([#7929]) - net: fix `GET_BUF_SIZE` constant for `target_os = "android"` ([#7889]) - runtime: avoid redundant unpark in current_thread scheduler ([#7834]) - runtime: don't park in `current_thread` if `before_park` defers waker ([#7835]) - io: fix write readiness on ESP32 on short writes ([#7872]) - runtime: wake deferred tasks before entering `block_in_place` ([#7879]) - sync: drop rx waker when oneshot receiver is dropped ([#7886]) - runtime: fix double increment of `num_idle_threads` on shutdown ([#7910], [#7918], [#7922]) Unstable -------- - fs: check for io-uring opcode support ([#7815]) - runtime: avoid lock acquisition after uring init ([#7850]) Documented ---------- - docs: update outdated unstable features section ([#7839]) - io: clarify the behavior of `AsyncWriteExt::shutdown()` ([#7908]) - io: explain how to flush stdout/stderr ([#7904]) - io: fix incorrect and confusing `AsyncWrite` documentation ([#7875]) - rt: clarify the documentation of `Runtime::spawn` ([#7803]) - rt: fix missing quotation in docs ([#7925]) - runtime: correct the default thread name in docs ([#7896]) - runtime: fix `event_interval` doc ([#7932]) - sync: clarify RwLock fairness documentation ([#7919]) - sync: clarify that `recv` returns `None` once closed and no more messages ([#7920]) - task: clarify when to use `spawn_blocking` vs dedicated threads ([#7923]) - task: doc that task drops before `JoinHandle` completion ([#7825]) - signal: guarantee that listeners never return `None` ([#7869]) - task: fix task module feature flags in docs ([#7891]) - task: fix two typos ([#7913]) - task: improve the docs of `Builder::spawn_local` ([#7828]) - time: add docs about auto-advance and when to use sleep ([#7858]) - util: fix typo in docs ([#7926]) [#7771]: tokio-rs/tokio#7771 [#7773]: tokio-rs/tokio#7773 [#7792]: tokio-rs/tokio#7792 [#7803]: tokio-rs/tokio#7803 [#7815]: tokio-rs/tokio#7815 [#7825]: tokio-rs/tokio#7825 [#7828]: tokio-rs/tokio#7828 [#7833]: tokio-rs/tokio#7833 [#7834]: tokio-rs/tokio#7834 [#7835]: tokio-rs/tokio#7835 [#7837]: tokio-rs/tokio#7837 [#7838]: tokio-rs/tokio#7838 [#7839]: tokio-rs/tokio#7839 [#7850]: tokio-rs/tokio#7850 [#7858]: tokio-rs/tokio#7858 [#7869]: tokio-rs/tokio#7869 [#7871]: tokio-rs/tokio#7871 [#7872]: tokio-rs/tokio#7872 [#7875]: tokio-rs/tokio#7875 [#7879]: tokio-rs/tokio#7879 [#7880]: tokio-rs/tokio#7880 [#7885]: tokio-rs/tokio#7885 [#7886]: tokio-rs/tokio#7886 [#7889]: tokio-rs/tokio#7889 [#7891]: tokio-rs/tokio#7891 [#7896]: tokio-rs/tokio#7896 [#7904]: tokio-rs/tokio#7904 [#7908]: tokio-rs/tokio#7908 [#7910]: tokio-rs/tokio#7910 [#7913]: tokio-rs/tokio#7913 [#7918]: tokio-rs/tokio#7918 [#7919]: tokio-rs/tokio#7919 [#7920]: tokio-rs/tokio#7920 [#7922]: tokio-rs/tokio#7922 [#7923]: tokio-rs/tokio#7923 [#7925]: tokio-rs/tokio#7925 [#7926]: tokio-rs/tokio#7926 [#7929]: tokio-rs/tokio#7929 [#7932]: tokio-rs/tokio#7932 Release notes: - 1.53.1: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.53.1 - 1.53.0: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.53.0 - 1.52.4: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.4 - 1.52.3: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.3 - 1.52.2: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.2 - 1.52.1: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.1 - 1.52.0: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0 - 1.51.4: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.4 - 1.51.3: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.3 - 1.51.2: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.2 - 1.51.1: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.1 - 1.51.0: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0 - 1.50.0: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.50.0 --- Transitive Dependencies ----------------------- - libc (0.2.180 -> 0.2.189) - mio (1.1.1 -> 1.2.3) - socket2 (0.6.2 -> 0.6.5) - syn 3.0.6 (added) - tokio-macros (2.6.0 -> 2.7.2) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Motivation
As discussed in #4730, #6315, seanmonstar/reqwest#1907,
private-attribution/ipa#1120, oxidecomputer/omicron#9619, and many other
issues, it is possible for the runtime to end up in a situation where
the only unparked worker thread to begin executing a long-running or
permanently blocking task, preventing the time and I/O drivers from
making progress and waking other tasks, since no other worker is
unparked to steal the driver(s). This can result in increased latency
for I/O and timer notifications, or in some cases, can permanently
deadlock the runtime.
Solution
This branch implements the solution I proposed four years ago in this
comment: if a worker has just returned from parking on the timer or
I/O driver and is about to poll a task from its run queue, it will
notify another parked worker to attempt to steal the driver. This should
reduce latency for dispatching I/O and timer notifications and reduce
the risk of deadlock. This is very similar to the approach taken in a
previous PR, #6251. The main difference here is that I've hidden this
behavior behind a runtime builder option,
enable_eager_driver_handoff(). I chose to do this because theincreased amount of cross-thread synchronization due to additional
worker thread wakeups has made an approach like this somewhat
controversial, and some users may prefer to avoid it despite the
potential benefits. Putting this behavior behind an unstable runtime
builder option allows those who want it to experiment with it, while not
committing to this approach as our permanent and canonical solution to
#6315. This was suggested by @carllerche in this comment on #6251.
I've also consolidated the wakeup triggered by a worker which had
previously parked on the timer or I/O driver with the wakeup from being
the last worker to
transition_from_searching, so we only trigger asingle unpark, rather than a second potentially spurious wakeup if both
conditions are true.
Finally, I've added a test which reproduces a (slightly contrived)
deadlock, which is prevented by eager driver handoff.