Skip to content

runtime: optional eager I/O driver/timer handoff when polling tasks - #8010

Merged
Darksonn merged 12 commits into
masterfrom
eliza/io-driver-heresy-mode
Apr 8, 2026
Merged

Darksonn merged 12 commits into
masterfrom
eliza/io-driver-heresy-mode

Conversation

@hawkw

@hawkw hawkw commented Apr 3, 2026

Copy link
Copy Markdown
Member

Motivation

As discussed in #4730, #6315, seanmonstar/reqwest#1907,
private-attribution/ipa#1120, oxidecomputer/omicron#9619, and many other
issues, it is possible for the runtime to end up in a situation where
the only unparked worker thread to begin executing a long-running or
permanently blocking task, preventing the time and I/O drivers from
making progress and waking other tasks, since no other worker is
unparked to steal the driver(s). This can result in increased latency
for I/O and timer notifications, or in some cases, can permanently
deadlock the runtime.

Solution

This branch implements the solution I proposed four years ago in this
comment
: if a worker has just returned from parking on the timer or
I/O driver and is about to poll a task from its run queue, it will
notify another parked worker to attempt to steal the driver. This should
reduce latency for dispatching I/O and timer notifications and reduce
the risk of deadlock. This is very similar to the approach taken in a
previous PR, #6251. The main difference here is that I've hidden this
behavior behind a runtime builder option,
enable_eager_driver_handoff(). I chose to do this because the
increased amount of cross-thread synchronization due to additional
worker thread wakeups has made an approach like this somewhat
controversial, and some users may prefer to avoid it despite the
potential benefits. Putting this behavior behind an unstable runtime
builder option allows those who want it to experiment with it, while not
committing to this approach as our permanent and canonical solution to
#6315. This was suggested by @carllerche in this comment on #6251.
I've also consolidated the wakeup triggered by a worker which had
previously parked on the timer or I/O driver with the wakeup from being
the last worker to transition_from_searching, so we only trigger a
single unpark, rather than a second potentially spurious wakeup if both
conditions are true.

Finally, I've added a test which reproduces a (slightly contrived)
deadlock, which is prevented by eager driver handoff.

@hawkw
hawkw requested a review from Darksonn April 3, 2026 20:56
@hawkw hawkw self-assigned this Apr 3, 2026
@hawkw hawkw added C-enhancement Category: A PR with an enhancement or bugfix. M-runtime Module: tokio/runtime labels Apr 3, 2026
@github-actions github-actions Bot added the R-loom-multi-thread Run loom multi-thread tests on this PR label Apr 3, 2026
@hawkw

hawkw commented Apr 3, 2026

Copy link
Copy Markdown
Member Author

Oh I really do not love that my reproducer apparently cannot make the runtime deadlock on ARM --- that suggests some difference of behavior there which is also disconcerting.

hawkw added 4 commits April 3, 2026 14:18
This way, it should be less susceptible to some platforms having slow
enough timers that the two tasks are woken in the same iteration of the
timer wheel, or whatever it is that broke CI.
@hawkw

hawkw commented Apr 3, 2026

Copy link
Copy Markdown
Member Author

Oh I really do not love that my reproducer apparently cannot make the runtime deadlock on ARM --- that suggests some difference of behavior there which is also disconcerting.

I suspect this was due to the test's previous reliance on timers; if the test was running in a sufficiently slow environment (such as a QEMU VM on a GitHub Acions runner), the two timers could conceivably complete in the same poll of the timer wheel, resulting in it "not deadlocking". I've rewritten the test to use Unix pipes instead which should hopefully be less flaky.

@Darksonn Darksonn left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementation looks pretty clean to me, thanks!

I'll leave this for a few days in case anyone else wants to review it, but otherwise I'll merge this soon.

@ongardie-atomix

Copy link
Copy Markdown

FWIW as a random user, thank you! I ran into the problem this fixes when testing a transaction system. My test got stuck with one task hogging one thread, even though many idle threads were available. I knew I had an unyielding retry loop (which I should fix), but I continued to dig in to explain how one task could wedge the entire runtime. The scenario was a single task in an unyielding retry loop, but a timer was set that should have unblocked another task that would get this one unstuck.

At least in my case, this PR solves the problem:

  • When I ran without enable_eager_driver_handoff, I saw a test failure on the 3rd attempt (and this was typical).
  • When I ran it with enable_eager_driver_handoff, I saw 100 successful test runs.

I think the PR makes sense conceptually and should make the runtime behavior more predictable for users. I can't speak to how it affects normal performance, however. I hope this can make its way to becoming easily enabled (without tokio_unstable) over time, and it seems like it might make a good default.

@hawkw

hawkw commented Apr 8, 2026

Copy link
Copy Markdown
Member Author

I think the PR makes sense conceptually and should make the runtime behavior more predictable for users. I can't speak to how it affects normal performance, however. I hope this can make its way to becoming easily enabled (without tokio_unstable) over time, and it seems like it might make a good default.

I would certainly like to expose this behavior without the unstable flag, and possibly as a default eventually: personally, I also feel that this behavior is probably correct for a majority of applications, and I expect it will probably result in better overall performance in most cases. However, I wanted to be a bit conservative when introducing a change to the runtime's behavior --- even if we confirm this to be an improvement in most cases, we may want to change the interface for enabling it, for example. I'm also hoping there's a path to dropping the unstable flag after it's been out for a couple releases.

@Darksonn

Darksonn commented Apr 8, 2026 •

Copy link
Copy Markdown
Member

I think if people try it out and confirm no serious perf regressions, we can just make this the default. It won't break any workflows.

It's a bit awkward taking into account that io-uring wants us to introduce per-thread IO contexts, which would bring back #6315 unless we make sure that threads have a way to check the ring owned by other threads for completions to avoid starvation. But probably this means that per-thread rings must be guarded behind a new runtime flavor and the default Tokio configuration will just have a single shared ring.

@hawkw

hawkw commented Apr 8, 2026

Copy link
Copy Markdown
Member Author

I think if people try it out and confirm no serious perf regressions, we can just make this the default. It won't break any workflows.

That sounds great, we could always end up changing the builder option to be disable_eager_driver_handoff for anyone who believes that this causes a performance regression for what they're doing, when we stabilize it.

@Darksonn
Darksonn merged commit fccc28f into master Apr 8, 2026
102 checks passed
@Darksonn
Darksonn deleted the eliza/io-driver-heresy-mode branch April 8, 2026 20:46
hawkw added a commit that referenced this pull request Apr 14, 2026
# 1.52.0 (April 14th, 2026)

## Added

- io: `AioSource::register_borrowed` for I/O safety support ([#7992])
- net: add `try_io` function to `unix::pipe` sender and receiver types
  ([#8030])

## Added (unstable)

- runtime: `Builder::enable_eager_driver_handoff` setting enable eager
  hand
  off of the I/O and time drivers before polling tasks ([#8010])
- taskdump: add `trace_with()` for customized task dumps ([#8025])
- taskdump: allow `impl FnMut()` in `trace_with`` instead of just `fn()`

  ([#8040])
- fs: support `io_uring` in `AsyncRead` for `File` ([#7907])

## Changed

- runtime: improve `spawn_blocking` scalability with sharded queue
  ([#7757])
- runtime: use `compare_exchange_weak()` in worker queue ([#8028])

## Fixed

- runtime: overflow second half of tasks when local queue is filled
  instead of
  first half ([#8029])

## Documented

- docs: fix typo in `oneshot::Sender::send` docs ([#8026])
- docs: hide #[tokio::main] attribute in the docs of `sync::watch`
  ([#8035])
- net: add docs on `ConnectionRefused` errors with UDP sockets ([#7870])

[#7757]: #7757
[#7870]: #7870
[#7907]: #7907
[#7992]: #7992
[#8010]: #8010
[#8025]: #8025
[#8026]: #8026
[#8028]: #8028
[#8029]: #8029
[#8030]: #8030
[#8035]: #8035
[#8040]: #8040
@hawkw hawkw mentioned this pull request Apr 14, 2026
hawkw added a commit that referenced this pull request Apr 14, 2026
# 1.52.0 (April 14th, 2026)

## Added

- io: `AioSource::register_borrowed` for I/O safety support ([#7992])
- net: add `try_io` function to `unix::pipe` sender and receiver types
  ([#8030])

## Added (unstable)

- runtime: `Builder::enable_eager_driver_handoff` setting enable eager
  hand off of the I/O and time drivers before polling tasks ([#8010])
- taskdump: add `trace_with()` for customized task dumps ([#8025])
- taskdump: allow `impl FnMut()` in `trace_with` instead of just `fn()`
  ([#8040])
- fs: support `io_uring` in `AsyncRead` for `File` ([#7907])

## Changed

- runtime: improve `spawn_blocking` scalability with sharded queue
  ([#7757])
- runtime: use `compare_exchange_weak()` in worker queue ([#8028])

## Fixed

- runtime: overflow second half of tasks when local queue is filled
  instead of first half ([#8029])

## Documented

- docs: fix typo in `oneshot::Sender::send` docs ([#8026])
- docs: hide #[tokio::main] attribute in the docs of `sync::watch`
  ([#8035])
- net: add docs on `ConnectionRefused` errors with UDP sockets ([#7870])

[#7757]: #7757
[#7870]: #7870
[#7907]: #7907
[#7992]: #7992
[#8010]: #8010
[#8025]: #8025
[#8026]: #8026
[#8028]: #8028
[#8029]: #8029
[#8030]: #8030
[#8035]: #8035
[#8040]: #8040
hawkw added a commit to oxidecomputer/oxide-tokio-rt that referenced this pull request Apr 14, 2026
This updates our minimum Tokio version to [1.52.0]. This allows us to
pick up two major fixes that will change our default runtime
configuration:

- tokio-rs/tokio#8010 (released in [1.52.0]) which fixes
  oxidecomputer/omicron#9619 when its builder option is enabled,
- tokio-rs/tokio#7431,(released in 1.51.0), which allows tasks in the
  LIFO slot to participate in work-stealing.

Subsequent commits will actually update our runtime settings after
picking up these releases.

[1.52.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0
[1.51.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0
hawkw added a commit to oxidecomputer/oxide-tokio-rt that referenced this pull request Apr 15, 2026
This updates our minimum Tokio version to [1.52.0]. This allows us to
pick up two major fixes that will change our default runtime
configuration:

- tokio-rs/tokio#8010 (released in [1.52.0]) which fixes
  oxidecomputer/omicron#9619 when its builder option is enabled,
- tokio-rs/tokio#7431,(released in 1.51.0), which allows tasks in the
  LIFO slot to participate in work-stealing.

Subsequent commits will actually update our runtime settings after
picking up these releases.

[1.52.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0
[1.51.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0
hawkw added a commit to oxidecomputer/oxide-tokio-rt that referenced this pull request Apr 15, 2026
This commit enables the eager I/O and time driver hand-off setting
for the multi-threaded Tokio runtime, which I added in
tokio-rs/tokio#8010. This is currently an experimental feature,
although it is your author's opinion that this is really a fix for
incorrect runtime behavior. It changes worker threads in the
multi-threaded runtime to wake another worker prior to polling tasks if
that worker had previously been parked on the I/O driver or timer
wheel. Eagerly handing off these resources should prevent pathologies
such as omicron#9619.
hawkw added a commit to oxidecomputer/oxide-tokio-rt that referenced this pull request Apr 15, 2026
This updates our minimum Tokio version to [1.52.0]. This allows us to
pick up two major fixes that will change our default runtime
configuration:

- tokio-rs/tokio#8010 (released in [1.52.0]) which fixes
  oxidecomputer/omicron#9619 when its builder option is enabled,
- tokio-rs/tokio#7431,(released in 1.51.0), which allows tasks in the
  LIFO slot to participate in work-stealing.

Subsequent commits will actually update our runtime settings after
picking up these releases.

[1.52.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0
[1.51.0]: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0
hawkw added a commit to oxidecomputer/oxide-tokio-rt that referenced this pull request Apr 15, 2026
This commit enables the eager I/O and time driver hand-off setting
for the multi-threaded Tokio runtime, which I added in
tokio-rs/tokio#8010. This is currently an experimental feature,
although it is your author's opinion that this is really a fix for
incorrect runtime behavior. It changes worker threads in the
multi-threaded runtime to wake another worker prior to polling tasks if
that worker had previously been parked on the I/O driver or timer
wheel. Eagerly handing off these resources should prevent pathologies
such as omicron#9619.
hawkw added a commit to oxidecomputer/omicron that referenced this pull request Apr 15, 2026
update `oxide-tokio-rt` to v0.1.4, `tokio` to v1.52.0

This branch updates our dependency on `oxide-tokio-rt` to pick up Tokio
v1.52.0 and the corresponding changes to the default runtime settings in
`oxide-tokio-rt`. In particular, this allows us to pick up two of my
upstream fixes in Tokio for a pair of issues that have been major thorns
in our side for some time:

* Tokio PR tokio-rs/tokio#7431, released in [Tokio v1.51.0],  changes
  the multi-threaded runtime to allow tasks in the LIFO slot to
  participate in work-stealing. Therefore, it should no longer be
  necessary to disable the LIFO slot optimization, as the pathology
  described in #8334, where a task placed in the LIFO slot can become
  permanently or semi-permanently stuck while the task that notified
  them runs for a long time without yielding, can no longer occur.
  `oxide-tokio-rt` v0.1.4 removes the runtime configuration to disable
  the LIFO slot as the issue has been fixed upstream.

* Tokio PR tokio-rs/tokio#8010, released in [Tokio v1.52.0], which adds
  eager handoff for the I/O and time drivers in the multi-threaded
  runtime. This is currently an experimental feature, although it is
  your author's opinion that this is really a fix for incorrect runtime
  behavior. It changes worker threads in the multi-threaded runtime to
  wake another worker prior to polling tasks if that worker had
  previously been parked on the I/O driver or timer wheel. Eagerly
  handing off these resources should prevent pathologies such as #9619.
  `oxide-tokio-rt` v0.1.4 enables this behavior by default.

Fixes #8334
Fixes #9619

[Tokio v1.52.0]:
https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0
[Tokio v1.51.0]:
https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0
hawkw added a commit to oxidecomputer/omicron that referenced this pull request Apr 15, 2026
This branch updates our dependency on `oxide-tokio-rt` to pick up Tokio
v1.52.0 and the corresponding changes to the default runtime settings in
`oxide-tokio-rt`. In particular, this allows us to pick up two of my
upstream fixes in Tokio for a pair of issues that have been major thorns
in our side for some time:

* Tokio PR tokio-rs/tokio#7431, released in [Tokio v1.51.0], changes the
  multi-threaded runtime to allow tasks in the LIFO slot to participate
  in work-stealing. Therefore, it should no longer be necessary to
  disable the LIFO slot optimization, as the pathology described in
  #8334, where a task placed in the LIFO slot can become permanently or
  semi-permanently stuck while the task that notified them runs for a
  long time without yielding, can no longer occur. `oxide-tokio-rt`
  v0.1.4 removes the runtime configuration to disable the LIFO slot as
  the issue has been fixed upstream.

* Tokio PR tokio-rs/tokio#8010, released in [Tokio v1.52.0], which adds
  eager handoff for the I/O and time drivers in the multi-threaded
  runtime. This is currently an experimental feature, although it is
  your author's opinion that this is really a fix for incorrect runtime
  behavior.[^1] It changes worker threads in the multi-threaded runtime
  to wake another worker prior to polling tasks if that worker had
  previously been parked on the I/O driver or timer wheel. Eagerly
  handing off these resources should prevent pathologies such as #9619.
  `oxide-tokio-rt` v0.1.4 enables this behavior by default.

Fixes #8334
Fixes #9619

[Tokio v1.52.0]:
  https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0
[Tokio v1.51.0]:
  https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0

[^1]: Cue @leftwo's dictum that a "performance regression" from fixing
      incorrect behavior...isn't a performance regression.
hawkw added a commit to oxidecomputer/omicron that referenced this pull request Apr 16, 2026
Commit #10272 updated our dependency on `oxide-tokio-rt` to v0.1.4 and
our `tokio` dependency to v1.52.0. This allowed us to pick up two of my
fixes for Tokio issues that have been a thorn in our side for a long
time, tokio-rs/tokio#7431 and tokio-rs/tokio#8010, which fix #8334 and
#9619, respectively. The nature of these fixes is described in greater
detail in #10272.

Unfortunately, #10272 had to be reverted (in #10279), since @iliana
discovered an unrelated regression in Tokio v1.52.0, tokio-rs/tokio#8056
(our issue #10277). This regression caused `spawn_blocking` to
occasionally hang, and was introduced in
tokio-rs/tokio@1604bc3 (PR
tokio-rs/tokio#7757).

I've since reverted this change upstream (tokio-rs/tokio#8057), and
published a patch release ([v1.52.1]), which fixes the regression.
Therefore, it is now once again safe to update our Tokio dependency to
pick up the other fixes. This commit does that. I've also confirmed that
the issue described in #10277 is no longer present in Tokio v1.52.1, as
demonstrated by the fact `cargo nextest run -p omicron-sled-agent
--stress-count 100 -- --exact artifact_store::test::issue_7796` now
succeeds without hanging once again.

Fixes #10272
eleboucher pushed a commit to eleboucher/towonel that referenced this pull request Apr 18, 2026
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [tokio](https://tokio.rs) ([source](https://github.com/tokio-rs/tokio)) | workspace.dependencies | minor | `1.51.1` → `1.52.1` |

---

### Release Notes

<details>
<summary>tokio-rs/tokio (tokio)</summary>

### [`v1.52.1`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.1): Tokio v1.52.1

[Compare Source](tokio-rs/tokio@tokio-1.52.0...tokio-1.52.1)

### 1.52.1 (April 16th, 2026)

#### Fixed

- runtime: revert [#&#8203;7757] to fix [a regression][#&#8203;8056] that causes `spawn_blocking` to hang ([#&#8203;8057])

[#&#8203;7757]: tokio-rs/tokio#7757

[#&#8203;8056]: tokio-rs/tokio#8056

[#&#8203;8057]: tokio-rs/tokio#8057

### [`v1.52.0`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0): Tokio v1.52.0

[Compare Source](tokio-rs/tokio@tokio-1.51.1...tokio-1.52.0)

### 1.52.0 (April 14th, 2026)

#### Added

- io: `AioSource::register_borrowed` for I/O safety support ([#&#8203;7992])
- net: add `try_io` function to `unix::pipe` sender and receiver types ([#&#8203;8030])

#### Added (unstable)

- runtime: `Builder::enable_eager_driver_handoff` setting enable eager hand off of the I/O and time drivers before polling tasks ([#&#8203;8010])
- taskdump: add `trace_with()` for customized task dumps ([#&#8203;8025])
- taskdump: allow `impl FnMut()` in `trace_with` instead of just `fn()` ([#&#8203;8040])
- fs: support `io_uring` in `AsyncRead` for `File` ([#&#8203;7907])

#### Changed

- runtime: improve `spawn_blocking` scalability with sharded queue ([#&#8203;7757])
- runtime: use `compare_exchange_weak()` in worker queue ([#&#8203;8028])

#### Fixed

- runtime: overflow second half of tasks when local queue is filled instead of first half ([#&#8203;8029])

#### Documented

- docs: fix typo in `oneshot::Sender::send` docs ([#&#8203;8026])
- docs: hide #\[tokio::main] attribute in the docs of `sync::watch` ([#&#8203;8035])
- net: add docs on `ConnectionRefused` errors with UDP sockets ([#&#8203;7870])

[#&#8203;7757]: tokio-rs/tokio#7757

[#&#8203;7870]: tokio-rs/tokio#7870

[#&#8203;7907]: tokio-rs/tokio#7907

[#&#8203;7992]: tokio-rs/tokio#7992

[#&#8203;8010]: tokio-rs/tokio#8010

[#&#8203;8025]: tokio-rs/tokio#8025

[#&#8203;8026]: tokio-rs/tokio#8026

[#&#8203;8028]: tokio-rs/tokio#8028

[#&#8203;8029]: tokio-rs/tokio#8029

[#&#8203;8030]: tokio-rs/tokio#8030

[#&#8203;8035]: tokio-rs/tokio#8035

[#&#8203;8040]: tokio-rs/tokio#8040

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xMDEuMSIsInVwZGF0ZWRJblZlciI6IjQzLjEwMS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJ0eXBlL21pbm9yIl19-->

Reviewed-on: https://git.erwanleboucher.dev/eleboucher/towonel/pulls/11
hawkw added a commit that referenced this pull request May 5, 2026
In order to re-enable stealing tasks from the LIFO slot, we would like
to allow the user to configure whether or not pushing tasks to the LIFO
slot unparks another worker thread. This is in order to avoid
performance regressions due to the overhead of additional cross-thread
wakeups (i.e. #8065). In [this comment][1] on #8092, @carllerche
suggested that we do this by having a single option controlling whether
we wake parked workers more or less aggressively which would also
control the eager I/O and time driver handoff behavior from #8010, which
also causes the runtime to wake parked workers more aggressively in
order to prevent potential deadlocks and task starvation.

This PR adds such a knob to the `runtime::Builder` type, which requires
`tokio_unstable`. This is represented as an `UnparkingMode` enum. We use
an enum rather than a `bool` primarily because I would like to add a
third unparking mode in the future. This mode would wake parked workers
much less frequently, but would instead track whether there are any
stealable tasks enqueued, and wake a stealer only when _transitioning_
from 0-1 stealable tasks, and have parking workers park with a timeout
while we are in the "stealable tasks" state. The idea is that we would
prevent deadlocks while avoiding spurious IPIs in this case. But, I
haven't actually implemented this behavior yet, so presently, the enum
has two variants: `Traditional`, which is the current runtime behavior,
and `Cautious`, which wakes more aggressively to try to prevent
deadlocks (and enables waking on LIFO pushes and on I/O driver handoff).
The existing `enable_eager_driver_handoff` is deprecated in favor of the
new API.

I also felt that the enum made it a bit easier to document the different
behaviors. If anyone has input on how we can better explain the two
unparking modes to users, I'd welcome suggestions!

Closes #8118
[1]: #8092 (review)
hawkw added a commit that referenced this pull request May 5, 2026
In order to re-enable stealing tasks from the LIFO slot, we would like
to allow the user to configure whether or not pushing tasks to the LIFO
slot unparks another worker thread. This is in order to avoid
performance regressions due to the overhead of additional cross-thread
wakeups (i.e. #8065). In [this comment][1] on #8092, @carllerche
suggested that we do this by having a single option controlling whether
we wake parked workers more or less aggressively which would also
control the eager I/O and time driver handoff behavior from #8010, which
also causes the runtime to wake parked workers more aggressively in
order to prevent potential deadlocks and task starvation.

This PR adds such a knob to the `runtime::Builder` type, which requires
`tokio_unstable`. This is represented as an `UnparkingMode` enum. We use
an enum rather than a `bool` primarily because I would like to add a
third unparking mode in the future. This mode would wake parked workers
much less frequently, but would instead track whether there are any
stealable tasks enqueued, and wake a stealer only when _transitioning_
from 0-1 stealable tasks, and have parking workers park with a timeout
while we are in the "stealable tasks" state. The idea is that we would
prevent deadlocks while avoiding spurious IPIs in this case. But, I
haven't actually implemented this behavior yet, so presently, the enum
has two variants: `Traditional`, which is the current runtime behavior,
and `Cautious`, which wakes more aggressively to try to prevent
deadlocks (and enables waking on LIFO pushes and on I/O driver handoff).
The existing `enable_eager_driver_handoff` is deprecated in favor of the
new API.

I also felt that the enum made it a bit easier to document the different
behaviors. If anyone has input on how we can better explain the two
unparking modes to users, I'd welcome suggestions!

Closes #8118
[1]: #8092 (review)
ahacop added a commit to ahacop/erwindb that referenced this pull request Sep 28, 2026
[cb skip]

Changes in 1.53.1
-----------------

1.53.1 (July 20th, 2026)
------------------------

Fixed
-----

- signal: restore MSRV by removing `OnceLock::wait` from the Windows handler ([#8300])

Fixed (unstable)
----------------

- time: fix alt timer cancellation and insertion race ([#8252])

Documented
----------

- runtime: remove dead link definition in Runtime::block_on ([#8301])

[#8252]: tokio-rs/tokio#8252
[#8300]: tokio-rs/tokio#8300
[#8301]: tokio-rs/tokio#8301

Changes in 1.53.0
-----------------

1.53.0 (July 17th, 2026)
------------------------

Added
-----

- fs: implement `From<OwnedFd>` and `From<OwnedHandle>` for `File` ([#8266])
- metrics: add task schedule latency metric ([#7986])
- net: add `SocketAddr` methods to Unix sockets ([#8144])

Changed
-------

- io: add `#[inline]` to IO trait impls for in-memory types ([#8242])
- net: implement UCred::pid on FreeBSD ([#8086])
- net: support Nuttx target os ([#8259])
- signal: refactor global variables on Windows ([#8231])
- sync: `mpsc::{Receiver,UnboundedReceiver}` now drops waker on drop, even if there are still senders ([#8095])
- taskdump: support taskdumps on s390x ([#8192])
- time: add `#[track_caller]` to `timeout_at()` ([#8077])
- time: consolidate mutex locks on spurious poll ([#8124])
- time: defer waker clone on spurious poll ([#8107])
- time: move lazy-registration state into `Sleep` ([#8132])
- tracing: remove unnecessary span clone ([#8126])

Fixed
-----

- io: do not treat zero-length reads as EOF in `Chain` ([#8251])
- net: use getpeereid for QNX peer credentials ([#8270])
- runtime: avoid illegal state in `FastRand` ([#8078])
- sync: wake mpsc receiver when a queued `reserve[_many]` returns permits ([#8260])
- taskdump: skip double wake on `Trace::capture`/`Trace::trace_with` ([#8043])
- time: avoid stack overflow in runtime constructor ([#8093])
- time (alt timer): ensure timers stay in the same runtime after `.reset()` ([#8169])

IO uring (unstable)
-------------------

- fs: use io-uring for `fs::try_exists` ([#8080])
- fs: use io-uring for renaming files ([#7800])
- rt: flush io-uring CQE in case of CQE overflow ([#8277])

Documented
----------

- docs: clarify cancel safety wording ([#8181])
- fs: clarify `create_dir_all` succeeds if path exists ([#8149])
- io: add warning about stdout reordering with multiple handles ([#8276])
- net: document pipe `try_read*`/`try_write*` readiness behavior ([#8032])
- runtime: document interaction with fork() ([#8202])
- sync: clarify broadcast lagging semantics ([#8239])
- sync: document memory ordering guarantees for Semaphore ([#8119])
- task: explain why `yield_now` defers its waker ([#8254])
- time: add panic docs to `timeout_at()` ([#8077])
- time: fix reversed poll order in timeout doc ([#8214])

[#7800]: tokio-rs/tokio#7800
[#7986]: tokio-rs/tokio#7986
[#8032]: tokio-rs/tokio#8032
[#8043]: tokio-rs/tokio#8043
[#8077]: tokio-rs/tokio#8077
[#8078]: tokio-rs/tokio#8078
[#8080]: tokio-rs/tokio#8080
[#8086]: tokio-rs/tokio#8086
[#8093]: tokio-rs/tokio#8093
[#8095]: tokio-rs/tokio#8095
[#8107]: tokio-rs/tokio#8107
[#8119]: tokio-rs/tokio#8119
[#8124]: tokio-rs/tokio#8124
[#8126]: tokio-rs/tokio#8126
[#8132]: tokio-rs/tokio#8132
[#8144]: tokio-rs/tokio#8144
[#8149]: tokio-rs/tokio#8149
[#8169]: tokio-rs/tokio#8169
[#8181]: tokio-rs/tokio#8181
[#8192]: tokio-rs/tokio#8192
[#8193]: tokio-rs/tokio#8193
[#8202]: tokio-rs/tokio#8202
[#8214]: tokio-rs/tokio#8214
[#8231]: tokio-rs/tokio#8231
[#8239]: tokio-rs/tokio#8239
[#8242]: tokio-rs/tokio#8242
[#8251]: tokio-rs/tokio#8251
[#8254]: tokio-rs/tokio#8254
[#8259]: tokio-rs/tokio#8259
[#8260]: tokio-rs/tokio#8260
[#8266]: tokio-rs/tokio#8266
[#8270]: tokio-rs/tokio#8270
[#8276]: tokio-rs/tokio#8276
[#8277]: tokio-rs/tokio#8277

Changes in 1.52.4
-----------------

1.52.4 (July 16th, 2026)
------------------------

Fixed
-----

- runtime: don't skip the driver when `before_park` schedules work ([#8222])

Fixed (unstable)
----------------

- taskdump: remove crate disambiguators from output ([#8264])

[#8264]: tokio-rs/tokio#8264
[#8222]: tokio-rs/tokio#8222

Changes in 1.52.3
-----------------

1.52.3 (May 8th, 2026)
----------------------

Fixed
-----

* sync: fix underflow in mpsc channel `len()` ([#8062])
* sync: notify receivers in mpsc `OwnedPermit::release()` method ([#8075])
* sync: require that an `RwLock` has `max_readers != 0` ([#8076])
* sync: return `Empty` from `try_recv()` when mpsc is closed with outstanding permits ([#8074])

[#8062]: tokio-rs/tokio#8062
[#8074]: tokio-rs/tokio#8074
[#8075]: tokio-rs/tokio#8075
[#8076]: tokio-rs/tokio#8076

Changes in 1.52.2
-----------------

1.52.2 (May 4th, 2026)
----------------------

This release reverts the LIFO slot stealing change introduced in 1.51.0 ([#7431]), due to [its performance impact][#8065]. ([#8100])

[#7431]: tokio-rs/tokio#7431
[#8065]: tokio-rs/tokio#8065
[#8100]: tokio-rs/tokio#8100

Changes in 1.52.1
-----------------

1.52.1 (April 16th, 2026)
-------------------------

Fixed
-----

- runtime: revert [#7757] to fix [a regression][#8056] that causes `spawn_blocking` to hang ([#8057])

[#7757]: tokio-rs/tokio#7757
[#8056]: tokio-rs/tokio#8056
[#8057]: tokio-rs/tokio#8057

Changes in 1.52.0
-----------------

1.52.0 (April 14th, 2026)
-------------------------

Added
-----

- io: `AioSource::register_borrowed` for I/O safety support ([#7992])
- net: add `try_io` function to `unix::pipe` sender and receiver types ([#8030])

Added (unstable)
----------------

- runtime: `Builder::enable_eager_driver_handoff` setting enable eager hand off of the I/O and time drivers before polling tasks ([#8010])
- taskdump: add `trace_with()` for customized task dumps ([#8025])
- taskdump: allow `impl FnMut()` in `trace_with` instead of just `fn()` ([#8040])
- fs: support `io_uring` in `AsyncRead` for `File` ([#7907])

Changed
-------

- runtime: improve `spawn_blocking` scalability with sharded queue ([#7757])
- runtime: use `compare_exchange_weak()` in worker queue ([#8028])

Fixed
-----

- runtime: overflow second half of tasks when local queue is filled instead of first half ([#8029])

Documented
----------

- docs: fix typo in `oneshot::Sender::send` docs ([#8026])
- docs: hide #[tokio::main] attribute in the docs of `sync::watch` ([#8035])
- net: add docs on `ConnectionRefused` errors with UDP sockets ([#7870])

[#7757]: tokio-rs/tokio#7757
[#7870]: tokio-rs/tokio#7870
[#7907]: tokio-rs/tokio#7907
[#7992]: tokio-rs/tokio#7992
[#8010]: tokio-rs/tokio#8010
[#8025]: tokio-rs/tokio#8025
[#8026]: tokio-rs/tokio#8026
[#8028]: tokio-rs/tokio#8028
[#8029]: tokio-rs/tokio#8029
[#8030]: tokio-rs/tokio#8030
[#8035]: tokio-rs/tokio#8035
[#8040]: tokio-rs/tokio#8040

Changes in 1.51.4
-----------------

1.51.4 (July 16th, 2026)
------------------------

Fixed
-----

- runtime: don't skip the driver when `before_park` schedules work ([#8222])

[#8222]: tokio-rs/tokio#8222

Changes in 1.51.3
-----------------

1.51.3 (May 8th, 2026)
----------------------

Fixed
-----

* sync: fix underflow in mpsc channel `len()` ([#8062])
* sync: notify receivers in mpsc `OwnedPermit::release()` method ([#8075])
* sync: require that an `RwLock` has `max_readers != 0` ([#8076])
* sync: return `Empty` from `try_recv()` when mpsc is closed with outstanding permits ([#8074])

[#8062]: tokio-rs/tokio#8062
[#8074]: tokio-rs/tokio#8074
[#8075]: tokio-rs/tokio#8075
[#8076]: tokio-rs/tokio#8076

Changes in 1.51.2
-----------------

1.51.2 (May 4th, 2026)
----------------------

This release reverts the LIFO slot stealing change introduced in 1.51.0 ([#7431]), due to [its performance impact][#8065]. ([#8100])

[#7431]: tokio-rs/tokio#7431
[#8065]: tokio-rs/tokio#8065
[#8100]: tokio-rs/tokio#8100

Changes in 1.51.1
-----------------

1.51.1 (April 8th, 2026)
------------------------

Fixed
-----

- sync: fix semaphore reopens after forget ([#8021])
- net: surface errors from `SO_ERROR` on `recv` for UDP sockets on Linux ([#8001])

Fixed (unstable)
----------------

- metrics: fix `worker_local_schedule_count` test ([#8008])
- rt: do not leak fd when cancelling io\_uring open operation ([#7983])

[#7983]: tokio-rs/tokio#7983
[#8001]: tokio-rs/tokio#8001
[#8008]: tokio-rs/tokio#8008
[#8021]: tokio-rs/tokio#8021

Changes in 1.51.0
-----------------

1.51.0 (April 3rd, 2026)
------------------------

Added
-----

- net: implement `get_peer_cred` on Hurd ([#7989])
- runtime: add `tokio::runtime::worker_index()` ([#7921])
- runtime: add runtime name ([#7924])
- runtime: stabilize `LocalRuntime` ([#7557])
- wasm: add wasm32-wasip2 networking support ([#7933])

Changed
-------

- runtime: steal tasks from the LIFO slot ([#7431])

Fixed
-----

- docs: do not show "Available on non-loom only." doc label ([#7977])
- macros: improve overall macro hygiene ([#7997])
- sync: fix `notify_waiters` priority in `Notify` ([#7996])
- sync: fix panic in `Chan::recv_many` when called with non-empty vector on closed channel ([#7991])

[#7431]: tokio-rs/tokio#7431
[#7557]: tokio-rs/tokio#7557
[#7921]: tokio-rs/tokio#7921
[#7924]: tokio-rs/tokio#7924
[#7933]: tokio-rs/tokio#7933
[#7977]: tokio-rs/tokio#7977
[#7989]: tokio-rs/tokio#7989
[#7991]: tokio-rs/tokio#7991
[#7996]: tokio-rs/tokio#7996
[#7997]: tokio-rs/tokio#7997

Changes in 1.50.0
-----------------

1.50.0 (Mar 3rd, 2026)
----------------------

Added
-----

- net: add `TcpStream::set_zero_linger` ([#7837])
- rt: add `is_rt_shutdown_err` ([#7771])

Changed
-------

- io: add optimizer hint that `memchr` returns in-bounds pointer ([#7792])
- io: implement vectored writes for `write_buf` ([#7871])
- runtime: panic when `event_interval` is set to 0 ([#7838])
- runtime: shorten default thread name to fit in Linux limit ([#7880])
- signal: remember the result of `SetConsoleCtrlHandler` ([#7833])
- signal: specialize windows `Registry` ([#7885])

Fixed
-----

- io: always cleanup `AsyncFd` registration list on deregister ([#7773])
- macros: remove (most) local `use` declarations in `tokio::select!` ([#7929])
- net: fix `GET_BUF_SIZE` constant for `target_os = "android"` ([#7889])
- runtime: avoid redundant unpark in current_thread scheduler ([#7834])
- runtime: don't park in `current_thread` if `before_park` defers waker ([#7835])
- io: fix write readiness on ESP32 on short writes ([#7872])
- runtime: wake deferred tasks before entering `block_in_place` ([#7879])
- sync: drop rx waker when oneshot receiver is dropped ([#7886])
- runtime: fix double increment of `num_idle_threads` on shutdown ([#7910], [#7918], [#7922])

Unstable
--------

- fs: check for io-uring opcode support ([#7815])
- runtime: avoid lock acquisition after uring init ([#7850])

Documented
----------

- docs: update outdated unstable features section ([#7839])
- io: clarify the behavior of `AsyncWriteExt::shutdown()` ([#7908])
- io: explain how to flush stdout/stderr ([#7904])
- io: fix incorrect and confusing `AsyncWrite` documentation ([#7875])
- rt: clarify the documentation of `Runtime::spawn` ([#7803])
- rt: fix missing quotation in docs ([#7925])
- runtime: correct the default thread name in docs ([#7896])
- runtime: fix `event_interval` doc ([#7932])
- sync: clarify RwLock fairness documentation ([#7919])
- sync: clarify that `recv` returns `None` once closed and no more messages ([#7920])
- task: clarify when to use `spawn_blocking` vs dedicated threads ([#7923])
- task: doc that task drops before `JoinHandle` completion ([#7825])
- signal: guarantee that listeners never return `None` ([#7869])
- task: fix task module feature flags in docs ([#7891])
- task: fix two typos ([#7913])
- task: improve the docs of `Builder::spawn_local` ([#7828])
- time: add docs about auto-advance and when to use sleep ([#7858])
- util: fix typo in docs ([#7926])

[#7771]: tokio-rs/tokio#7771
[#7773]: tokio-rs/tokio#7773
[#7792]: tokio-rs/tokio#7792
[#7803]: tokio-rs/tokio#7803
[#7815]: tokio-rs/tokio#7815
[#7825]: tokio-rs/tokio#7825
[#7828]: tokio-rs/tokio#7828
[#7833]: tokio-rs/tokio#7833
[#7834]: tokio-rs/tokio#7834
[#7835]: tokio-rs/tokio#7835
[#7837]: tokio-rs/tokio#7837
[#7838]: tokio-rs/tokio#7838
[#7839]: tokio-rs/tokio#7839
[#7850]: tokio-rs/tokio#7850
[#7858]: tokio-rs/tokio#7858
[#7869]: tokio-rs/tokio#7869
[#7871]: tokio-rs/tokio#7871
[#7872]: tokio-rs/tokio#7872
[#7875]: tokio-rs/tokio#7875
[#7879]: tokio-rs/tokio#7879
[#7880]: tokio-rs/tokio#7880
[#7885]: tokio-rs/tokio#7885
[#7886]: tokio-rs/tokio#7886
[#7889]: tokio-rs/tokio#7889
[#7891]: tokio-rs/tokio#7891
[#7896]: tokio-rs/tokio#7896
[#7904]: tokio-rs/tokio#7904
[#7908]: tokio-rs/tokio#7908
[#7910]: tokio-rs/tokio#7910
[#7913]: tokio-rs/tokio#7913
[#7918]: tokio-rs/tokio#7918
[#7919]: tokio-rs/tokio#7919
[#7920]: tokio-rs/tokio#7920
[#7922]: tokio-rs/tokio#7922
[#7923]: tokio-rs/tokio#7923
[#7925]: tokio-rs/tokio#7925
[#7926]: tokio-rs/tokio#7926
[#7929]: tokio-rs/tokio#7929
[#7932]: tokio-rs/tokio#7932

Release notes:
- 1.53.1: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.53.1
- 1.53.0: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.53.0
- 1.52.4: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.4
- 1.52.3: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.3
- 1.52.2: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.2
- 1.52.1: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.1
- 1.52.0: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0
- 1.51.4: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.4
- 1.51.3: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.3
- 1.51.2: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.2
- 1.51.1: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.1
- 1.51.0: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0
- 1.50.0: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.50.0

---

Transitive Dependencies
-----------------------

- libc (0.2.180 -> 0.2.189)
- mio (1.1.1 -> 1.2.3)
- socket2 (0.6.2 -> 0.6.5)
- syn 3.0.6 (added)
- tokio-macros (2.6.0 -> 2.7.2)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-tokio Area: The main tokio crate C-enhancement Category: A PR with an enhancement or bugfix. M-runtime Module: tokio/runtime R-loom-multi-thread Run loom multi-thread tests on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants