Skip to content

runtime: steal tasks from the LIFO slot - #7431

Merged
Darksonn merged 26 commits into
masterfrom
eliza/lifo-steal
Apr 3, 2026
Merged

Darksonn merged 26 commits into
masterfrom
eliza/lifo-steal

Conversation

@hawkw

@hawkw hawkw commented Jun 27, 2025 •

Copy link
Copy Markdown
Member

Motivation

Worker threads in the multi-threaded runtime include a per-worker LIFO
slot which stores the last task notified by another task running on that
worker. This allows the last-notified task to be polled first when the
currently running task completes, decreasing latency in message-passing
"ping-pong" scenarios.

However --- as described in #4941 --- there's an issue with this that
can cause severe problems in some situations: the task in the LIFO slot
cannot be stolen by a work-stealing worker thread. This means that if a
task notifies another task and then goes CPU bound for a long period of
time without yielding, the notified task will never be able to execute
until the task that notified it can yield. This can result in a very
severe latency bubble in some scenarios. See, for instance, #4323,
#6954, oxidecomputer/omicron#8334, etc.

As a workaround, PR #4936 added an unstable runtime::Builder option
to disable the LIFO slot
. However, this is a less-than-ideal
solution, as it means that applications which disable the LIFO slot due
to occasional usage patterns that cause latency bubbles when it is
enabled cannot benefit from the potential latency improvements it
offers in other usage patterns. And, it's an unstable option which the
user has to discover. In most cases, people whose programs contain usage
patterns that are pathological with regards to the LIFO slot don't know
this ahead of time: the typical narrative is that you write code that
happens to follow such a pattern, discover an unexpected latency spike
or hang in production, and then learn how to disable the LIFO slot. It
would be much nicer if the task in the LIFO slot could participate in
work-stealing like every other task.

Solution

This branch makes tasks in the LIFO slot stealable.

Broadly, I've taken the following approach:

  1. Add a new AtomicNotified type that implements an atomically
    swappable cell storing a Notified task
    (1220253), and use it to represent
    the LIFO slot instead of an Option<Notified<Arc<Handle>>>. This
    way, other workers can take a task out of the LIFO slot while
    work-stealing.

  2. Move the LIFO slot out of the worker::Core struct and into the run
    queue's Inner type (75d8116),
    making it shared state between the Local side of the queue owned by
    the worker itself and the Steal side used by remote workers to
    steal from the queue.

    There's a bunch of additional code in worker::Core for managing
    whether to atually run a task from the LIFO slot or not. I opted
    not to move any of this code into the run queue itself, as it
    depends on other bits of internal worker state. Instead, we just
    expose to the worker separate APIs for pushing/popping to/from the
    main queue and for pushing/popping to/from the LIFO slot, resulting
    in a fairly small diff to the worker's run loop.

  3. Change the work-stealing code to also steal the LIFO task
    (730a581 and
    cb27dda). This actually turned out
    to be pretty straightforwards: once we've stolen a chunk of tasks
    from the targeted worker's run queue, we now also grab whatever's in
    its' LIFO slot as well. If we stole a LIFO task, it's returned from
    the steal_into method in lieu of the first task in the run queue, so
    that it gets to execute first, maintaining the latency improvement for
    recently-notified tasks. This also was simpler than trying to wedge it
    into the chunk of tasks to be pushed to the stealer's queue.

I've made the following test changes while working on this branch:

  • Added an integration test that reproduces a scenario where a task in
    the LIFO slot is blocked from running when a task notifies it and then
    blocks indefinitely. I've confirmed that this test fails on master
    and passes on this branch.
  • Added new Loom tests for work-stealing involving the LIFO slot.
    These are in addition to the existing work-stealing Loom tests, as
    tasks notified by I/O or timers are still woken to the normal run
    queue.
  • A small change to the
    rt_unstable_metrics::worker_local_queue_depth integration test,
    which was necessary as tasks in the LIFO slot now "count" towards the
    worker's queue depth. We now have to make sure the no-op task that's
    spawned has completed before spawning the tasks we actually attempt to
    count, as it seems to sometimes end up in the LIFO slot and sometimes
    not, causing the test to flake out.

Fixes #4941

hawkw added 3 commits June 27, 2025 14:15
This commit changes the LIFO slot on multi-threaded runtime workers from
being a mutable `Option<Notified<Arc<Handle>>>` to a new
`AtomicNotified` type. This
type implements a cell containing a nullable task pointer which can
be swapped atomically. It's analogous to `AtomicCell` but with the extra
`PhantomData` to remember the task's scheduler type parameter, which
would otherwise be erased by the conversion into a `*mut Header``
pointer.

This change is in preparation for a subsequent change to allow
work-stealing from the LIFO slot (see: #4941).
This way, it's accesible by the stealer. Leave all the LIFO *accounting*
(i.e. deciding whether we hit the LIFO slot or not) up to the worker.

Gotta figure out whether the load of lifo presence will race...ugh.
@github-actions github-actions Bot added R-loom-current-thread Run loom current-thread tests on this PR R-loom-multi-thread Run loom multi-thread tests on this PR R-loom-multi-thread-alt labels Jun 27, 2025
@hawkw hawkw removed the R-loom-current-thread Run loom current-thread tests on this PR label Jun 30, 2025
@github-actions github-actions Bot added the R-loom-current-thread Run loom current-thread tests on this PR label Jun 30, 2025
This commit adds a test ensuring that if a task is notified to the LIFO
slot by another task which then blocks the worker thread forever, the
LIFO task is eventually stolen by another worker. I've confimed that
this test fails on the `master` branch, and passes after these changes.
hawkw and others added 5 commits June 30, 2025 12:53
This test spawns a task that sometimes ends up in the LIFO slot and
sometimes doesn't. This was previously fine as the LIFO slot didn't
count for `worker_local_queue_depth`, but now it does. Thus, we have to
make sure that task no longer exists before asserting about queue depth.
@hawkw hawkw changed the title [WIP] runtime: steal from the LIFO slot runtime: steal tasks from the LIFO slot Jul 1, 2025
@hawkw hawkw self-assigned this Jul 1, 2025
@hawkw hawkw added C-enhancement Category: A PR with an enhancement or bugfix. A-tokio Area: The main tokio crate M-runtime Module: tokio/runtime labels Jul 1, 2025
@hawkw
hawkw marked this pull request as ready for review July 1, 2025 19:39
@hawkw
hawkw requested review from ADD-SP, Darksonn and carllerche July 1, 2025 19:39
@hawkw

hawkw commented Jul 1, 2025

Copy link
Copy Markdown
Member Author

(shoutout to @mkeeter for nerd-sniping me into actually doing this)

Comment on lines +463 to +465
// If we also grabbed the task from the LIFO slot, include that in the
// steal count as well.
dst_stats.incr_steal_count(n as u16 + lifo.is_some() as u16);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For reviewers: Rust has defined the casting from bool to u16.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's also possible to use u16::from(bool), if you care about cast_lossless cleanliness.

hawkw added a commit to oxidecomputer/omicron that referenced this pull request Apr 15, 2026
This branch updates our dependency on `oxide-tokio-rt` to pick up Tokio
v1.52.0 and the corresponding changes to the default runtime settings in
`oxide-tokio-rt`. In particular, this allows us to pick up two of my
upstream fixes in Tokio for a pair of issues that have been major thorns
in our side for some time:

* Tokio PR tokio-rs/tokio#7431, released in [Tokio v1.51.0], changes the
  multi-threaded runtime to allow tasks in the LIFO slot to participate
  in work-stealing. Therefore, it should no longer be necessary to
  disable the LIFO slot optimization, as the pathology described in
  #8334, where a task placed in the LIFO slot can become permanently or
  semi-permanently stuck while the task that notified them runs for a
  long time without yielding, can no longer occur. `oxide-tokio-rt`
  v0.1.4 removes the runtime configuration to disable the LIFO slot as
  the issue has been fixed upstream.

* Tokio PR tokio-rs/tokio#8010, released in [Tokio v1.52.0], which adds
  eager handoff for the I/O and time drivers in the multi-threaded
  runtime. This is currently an experimental feature, although it is
  your author's opinion that this is really a fix for incorrect runtime
  behavior.[^1] It changes worker threads in the multi-threaded runtime
  to wake another worker prior to polling tasks if that worker had
  previously been parked on the I/O driver or timer wheel. Eagerly
  handing off these resources should prevent pathologies such as #9619.
  `oxide-tokio-rt` v0.1.4 enables this behavior by default.

Fixes #8334
Fixes #9619

[Tokio v1.52.0]:
  https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0
[Tokio v1.51.0]:
  https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0

[^1]: Cue @leftwo's dictum that a "performance regression" from fixing
      incorrect behavior...isn't a performance regression.
hawkw added a commit to oxidecomputer/omicron that referenced this pull request Apr 16, 2026
Commit #10272 updated our dependency on `oxide-tokio-rt` to v0.1.4 and
our `tokio` dependency to v1.52.0. This allowed us to pick up two of my
fixes for Tokio issues that have been a thorn in our side for a long
time, tokio-rs/tokio#7431 and tokio-rs/tokio#8010, which fix #8334 and
#9619, respectively. The nature of these fixes is described in greater
detail in #10272.

Unfortunately, #10272 had to be reverted (in #10279), since @iliana
discovered an unrelated regression in Tokio v1.52.0, tokio-rs/tokio#8056
(our issue #10277). This regression caused `spawn_blocking` to
occasionally hang, and was introduced in
tokio-rs/tokio@1604bc3 (PR
tokio-rs/tokio#7757).

I've since reverted this change upstream (tokio-rs/tokio#8057), and
published a patch release ([v1.52.1]), which fixes the regression.
Therefore, it is now once again safe to update our Tokio dependency to
pick up the other fixes. This commit does that. I've also confirmed that
the issue described in #10277 is no longer present in Tokio v1.52.1, as
demonstrated by the fact `cargo nextest run -p omicron-sled-agent
--stress-count 100 -- --exact artifact_store::test::issue_7796` now
succeeds without hanging once again.

Fixes #10272
hawkw added a commit that referenced this pull request Apr 20, 2026
Currently, the `rt_threaded::lifo_stealable` test I added in #7431
spawns an additional task which sleeps on a 4ms timer in a loop. This
ensures that no worker remains permanently parked. This was added
because it was necessary to stop the LIFO slot deadlock from occurring
prior to changes in the logic for determining whether to notify another
worker, which is what @Darksonn  was referring to in [this comment][1].
Removing the `churn()` test makes the test actually validate that
another worker is notified to steal the LIFO task, and that the changes
from #7431 will *always* prevent a LIFO slot deadlock, regardless of the
behavior of other tasks on the runtime. See also [this comment][2] for
further discussion.

[1]: #7431 (comment)
[2]: #8069 (comment)
Darksonn added a commit to Darksonn/tokio that referenced this pull request May 1, 2026
Darksonn added a commit to Darksonn/tokio that referenced this pull request May 2, 2026
Darksonn added a commit to Darksonn/tokio that referenced this pull request May 2, 2026
Darksonn added a commit to Darksonn/tokio that referenced this pull request May 2, 2026
eleboucher pushed a commit to eleboucher/towonel that referenced this pull request May 4, 2026
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [tokio](https://tokio.rs) ([source](https://github.com/tokio-rs/tokio)) | workspace.dependencies | patch | `1.52.1` → `1.52.2` |

---

### Release Notes

<details>
<summary>tokio-rs/tokio (tokio)</summary>

### [`v1.52.2`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.2): Tokio v1.52.2

[Compare Source](tokio-rs/tokio@tokio-1.52.1...tokio-1.52.2)

### 1.52.2 (May 4th, 2026)

This release reverts the LIFO slot stealing change introduced in 1.51.0 ([#&#8203;7431]), due to [its performance impact][#&#8203;8065]. ([#&#8203;8100])

[#&#8203;7431]: tokio-rs/tokio#7431

[#&#8203;8065]: tokio-rs/tokio#8065

[#&#8203;8100]: tokio-rs/tokio#8100

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xMDEuMSIsInVwZGF0ZWRJblZlciI6IjQzLjEwMS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJ0eXBlL3BhdGNoIl19-->

Reviewed-on: https://git.erwanleboucher.dev/eleboucher/towonel/pulls/31
hawkw added a commit to oxidecomputer/oxide-tokio-rt that referenced this pull request May 4, 2026
This reverts commit 66c189b, which
removed the code for disabling Tokio's LIFO slot optimization. We
re-enabled the LIFO slot since tokio-rs/tokio#7431 allowed it to
participate in work-stealing, and this change was released in Tokio
v1.51.0. However, Tokio v1.52.2 contained commit tokio-rs/tokio#8100,
which reverted tokio-rs/tokio#7431, due to a performance regression in
some applications (tokio-rs/tokio#8065). Therefore, this means that if
we update to Tokio v1.52.2, we will once again be susceptible to
LIFO-slot-related deadlocks. This commit re-disables the LIFO slot.

Hopefully, we will be able to figure out a solution upstream that allows
stealing LIFO tasks while reducing spurious cross-thread wakeups. But
for now, I think that we are best off leaving this disabled. Sigh.
hawkw added a commit to oxidecomputer/oxide-tokio-rt that referenced this pull request May 4, 2026
This reverts commit 66c189b, which
removed the code for disabling Tokio's LIFO slot optimization. We
re-enabled the LIFO slot since tokio-rs/tokio#7431 allowed it to
participate in work-stealing, and this change was released in Tokio
v1.51.0. However, Tokio v1.52.2 contained commit tokio-rs/tokio#8100,
which reverted tokio-rs/tokio#7431, due to a performance regression in
some applications (tokio-rs/tokio#8065). Therefore, this means that if
we update to Tokio v1.52.2, we will once again be susceptible to
LIFO-slot-related deadlocks. This commit re-disables the LIFO slot.

Hopefully, we will be able to figure out a solution upstream that allows
stealing LIFO tasks while reducing spurious cross-thread wakeups. But
for now, I think that we are best off leaving this disabled. Sigh.
hawkw added a commit that referenced this pull request May 5, 2026
This commit puts back the change to allow stealing tasks from the LIFO
slot, which was originally introduced in #7431. This change was reverted
in 9605999 in order to fix the
performance regression due to increased cross thread wakeups (#8065).
Previously, pushing a task to an unoccupied LIFO slot would not notify a
parked worker, but when #7431 made the LIFO slot stealable, we started
notifying in that case, which increases runtime overhead. Therefore, the
change was reverted in v1.52.2. This commit reintroduces it.
Subsequent commits on this branch will allow configuring whether LIFO
slot pushes wake a parked worker, so that we can allow stealing from
the LIFO slot while making the increased wakeup overhead opt-in.

Closes #4941
hawkw added a commit that referenced this pull request May 5, 2026
In #7431, I originally implemented `AtomicNotified` with the `AcqRel`
atomic ordering, which is all that *should* be necessary here. While
trying to debug a failing test on ARM, I changed it to `SeqCst`. This
fixed the test, but was not actually necessary to solve the root cause
of that test failure, which was ultimately fixed in #8008 instead. Using
`SeqCst` here and locking the bus probably increases the overhead of
using the LIFO slot substantially, so I've un-done that.
hawkw added a commit that referenced this pull request May 5, 2026
Currently, the `rt_threaded::lifo_stealable` test I added in #7431
spawns an additional task which sleeps on a 4ms timer in a loop. This
ensures that no worker remains permanently parked. This was added
because it was necessary to stop the LIFO slot deadlock from occurring
prior to changes in the logic for determining whether to notify another
worker, which is what @Darksonn  was referring to in [this comment][1].
Removing the `churn()` test makes the test actually validate that
another worker is notified to steal the LIFO task, and that the changes
from #7431 will *always* prevent a LIFO slot deadlock, regardless of the
behavior of other tasks on the runtime. See also [this comment][2] for
further discussion.

[1]: #7431 (comment)
[2]: #8069 (comment)
eleboucher pushed a commit to eleboucher/towonel that referenced this pull request May 20, 2026
This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [tokio](https://tokio.rs) ([source](https://github.com/tokio-rs/tokio)) | workspace.dependencies | patch | `1.52.1` → `1.52.2` |

---

### Release Notes

<details>
<summary>tokio-rs/tokio (tokio)</summary>

### [`v1.52.2`](https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.2): Tokio v1.52.2

[Compare Source](tokio-rs/tokio@tokio-1.52.1...tokio-1.52.2)

### 1.52.2 (May 4th, 2026)

This release reverts the LIFO slot stealing change introduced in 1.51.0 ([#&#8203;7431]), due to [its performance impact][#&#8203;8065]. ([#&#8203;8100])

[#&#8203;7431]: tokio-rs/tokio#7431

[#&#8203;8065]: tokio-rs/tokio#8065

[#&#8203;8100]: tokio-rs/tokio#8100

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xMDEuMSIsInVwZGF0ZWRJblZlciI6IjQzLjEwMS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJ0eXBlL3BhdGNoIl19-->

Reviewed-on: https://git.erwanleboucher.dev/eleboucher/towonel/pulls/31
ahacop added a commit to ahacop/erwindb that referenced this pull request Sep 28, 2026
[cb skip]

Changes in 1.53.1
-----------------

1.53.1 (July 20th, 2026)
------------------------

Fixed
-----

- signal: restore MSRV by removing `OnceLock::wait` from the Windows handler ([#8300])

Fixed (unstable)
----------------

- time: fix alt timer cancellation and insertion race ([#8252])

Documented
----------

- runtime: remove dead link definition in Runtime::block_on ([#8301])

[#8252]: tokio-rs/tokio#8252
[#8300]: tokio-rs/tokio#8300
[#8301]: tokio-rs/tokio#8301

Changes in 1.53.0
-----------------

1.53.0 (July 17th, 2026)
------------------------

Added
-----

- fs: implement `From<OwnedFd>` and `From<OwnedHandle>` for `File` ([#8266])
- metrics: add task schedule latency metric ([#7986])
- net: add `SocketAddr` methods to Unix sockets ([#8144])

Changed
-------

- io: add `#[inline]` to IO trait impls for in-memory types ([#8242])
- net: implement UCred::pid on FreeBSD ([#8086])
- net: support Nuttx target os ([#8259])
- signal: refactor global variables on Windows ([#8231])
- sync: `mpsc::{Receiver,UnboundedReceiver}` now drops waker on drop, even if there are still senders ([#8095])
- taskdump: support taskdumps on s390x ([#8192])
- time: add `#[track_caller]` to `timeout_at()` ([#8077])
- time: consolidate mutex locks on spurious poll ([#8124])
- time: defer waker clone on spurious poll ([#8107])
- time: move lazy-registration state into `Sleep` ([#8132])
- tracing: remove unnecessary span clone ([#8126])

Fixed
-----

- io: do not treat zero-length reads as EOF in `Chain` ([#8251])
- net: use getpeereid for QNX peer credentials ([#8270])
- runtime: avoid illegal state in `FastRand` ([#8078])
- sync: wake mpsc receiver when a queued `reserve[_many]` returns permits ([#8260])
- taskdump: skip double wake on `Trace::capture`/`Trace::trace_with` ([#8043])
- time: avoid stack overflow in runtime constructor ([#8093])
- time (alt timer): ensure timers stay in the same runtime after `.reset()` ([#8169])

IO uring (unstable)
-------------------

- fs: use io-uring for `fs::try_exists` ([#8080])
- fs: use io-uring for renaming files ([#7800])
- rt: flush io-uring CQE in case of CQE overflow ([#8277])

Documented
----------

- docs: clarify cancel safety wording ([#8181])
- fs: clarify `create_dir_all` succeeds if path exists ([#8149])
- io: add warning about stdout reordering with multiple handles ([#8276])
- net: document pipe `try_read*`/`try_write*` readiness behavior ([#8032])
- runtime: document interaction with fork() ([#8202])
- sync: clarify broadcast lagging semantics ([#8239])
- sync: document memory ordering guarantees for Semaphore ([#8119])
- task: explain why `yield_now` defers its waker ([#8254])
- time: add panic docs to `timeout_at()` ([#8077])
- time: fix reversed poll order in timeout doc ([#8214])

[#7800]: tokio-rs/tokio#7800
[#7986]: tokio-rs/tokio#7986
[#8032]: tokio-rs/tokio#8032
[#8043]: tokio-rs/tokio#8043
[#8077]: tokio-rs/tokio#8077
[#8078]: tokio-rs/tokio#8078
[#8080]: tokio-rs/tokio#8080
[#8086]: tokio-rs/tokio#8086
[#8093]: tokio-rs/tokio#8093
[#8095]: tokio-rs/tokio#8095
[#8107]: tokio-rs/tokio#8107
[#8119]: tokio-rs/tokio#8119
[#8124]: tokio-rs/tokio#8124
[#8126]: tokio-rs/tokio#8126
[#8132]: tokio-rs/tokio#8132
[#8144]: tokio-rs/tokio#8144
[#8149]: tokio-rs/tokio#8149
[#8169]: tokio-rs/tokio#8169
[#8181]: tokio-rs/tokio#8181
[#8192]: tokio-rs/tokio#8192
[#8193]: tokio-rs/tokio#8193
[#8202]: tokio-rs/tokio#8202
[#8214]: tokio-rs/tokio#8214
[#8231]: tokio-rs/tokio#8231
[#8239]: tokio-rs/tokio#8239
[#8242]: tokio-rs/tokio#8242
[#8251]: tokio-rs/tokio#8251
[#8254]: tokio-rs/tokio#8254
[#8259]: tokio-rs/tokio#8259
[#8260]: tokio-rs/tokio#8260
[#8266]: tokio-rs/tokio#8266
[#8270]: tokio-rs/tokio#8270
[#8276]: tokio-rs/tokio#8276
[#8277]: tokio-rs/tokio#8277

Changes in 1.52.4
-----------------

1.52.4 (July 16th, 2026)
------------------------

Fixed
-----

- runtime: don't skip the driver when `before_park` schedules work ([#8222])

Fixed (unstable)
----------------

- taskdump: remove crate disambiguators from output ([#8264])

[#8264]: tokio-rs/tokio#8264
[#8222]: tokio-rs/tokio#8222

Changes in 1.52.3
-----------------

1.52.3 (May 8th, 2026)
----------------------

Fixed
-----

* sync: fix underflow in mpsc channel `len()` ([#8062])
* sync: notify receivers in mpsc `OwnedPermit::release()` method ([#8075])
* sync: require that an `RwLock` has `max_readers != 0` ([#8076])
* sync: return `Empty` from `try_recv()` when mpsc is closed with outstanding permits ([#8074])

[#8062]: tokio-rs/tokio#8062
[#8074]: tokio-rs/tokio#8074
[#8075]: tokio-rs/tokio#8075
[#8076]: tokio-rs/tokio#8076

Changes in 1.52.2
-----------------

1.52.2 (May 4th, 2026)
----------------------

This release reverts the LIFO slot stealing change introduced in 1.51.0 ([#7431]), due to [its performance impact][#8065]. ([#8100])

[#7431]: tokio-rs/tokio#7431
[#8065]: tokio-rs/tokio#8065
[#8100]: tokio-rs/tokio#8100

Changes in 1.52.1
-----------------

1.52.1 (April 16th, 2026)
-------------------------

Fixed
-----

- runtime: revert [#7757] to fix [a regression][#8056] that causes `spawn_blocking` to hang ([#8057])

[#7757]: tokio-rs/tokio#7757
[#8056]: tokio-rs/tokio#8056
[#8057]: tokio-rs/tokio#8057

Changes in 1.52.0
-----------------

1.52.0 (April 14th, 2026)
-------------------------

Added
-----

- io: `AioSource::register_borrowed` for I/O safety support ([#7992])
- net: add `try_io` function to `unix::pipe` sender and receiver types ([#8030])

Added (unstable)
----------------

- runtime: `Builder::enable_eager_driver_handoff` setting enable eager hand off of the I/O and time drivers before polling tasks ([#8010])
- taskdump: add `trace_with()` for customized task dumps ([#8025])
- taskdump: allow `impl FnMut()` in `trace_with` instead of just `fn()` ([#8040])
- fs: support `io_uring` in `AsyncRead` for `File` ([#7907])

Changed
-------

- runtime: improve `spawn_blocking` scalability with sharded queue ([#7757])
- runtime: use `compare_exchange_weak()` in worker queue ([#8028])

Fixed
-----

- runtime: overflow second half of tasks when local queue is filled instead of first half ([#8029])

Documented
----------

- docs: fix typo in `oneshot::Sender::send` docs ([#8026])
- docs: hide #[tokio::main] attribute in the docs of `sync::watch` ([#8035])
- net: add docs on `ConnectionRefused` errors with UDP sockets ([#7870])

[#7757]: tokio-rs/tokio#7757
[#7870]: tokio-rs/tokio#7870
[#7907]: tokio-rs/tokio#7907
[#7992]: tokio-rs/tokio#7992
[#8010]: tokio-rs/tokio#8010
[#8025]: tokio-rs/tokio#8025
[#8026]: tokio-rs/tokio#8026
[#8028]: tokio-rs/tokio#8028
[#8029]: tokio-rs/tokio#8029
[#8030]: tokio-rs/tokio#8030
[#8035]: tokio-rs/tokio#8035
[#8040]: tokio-rs/tokio#8040

Changes in 1.51.4
-----------------

1.51.4 (July 16th, 2026)
------------------------

Fixed
-----

- runtime: don't skip the driver when `before_park` schedules work ([#8222])

[#8222]: tokio-rs/tokio#8222

Changes in 1.51.3
-----------------

1.51.3 (May 8th, 2026)
----------------------

Fixed
-----

* sync: fix underflow in mpsc channel `len()` ([#8062])
* sync: notify receivers in mpsc `OwnedPermit::release()` method ([#8075])
* sync: require that an `RwLock` has `max_readers != 0` ([#8076])
* sync: return `Empty` from `try_recv()` when mpsc is closed with outstanding permits ([#8074])

[#8062]: tokio-rs/tokio#8062
[#8074]: tokio-rs/tokio#8074
[#8075]: tokio-rs/tokio#8075
[#8076]: tokio-rs/tokio#8076

Changes in 1.51.2
-----------------

1.51.2 (May 4th, 2026)
----------------------

This release reverts the LIFO slot stealing change introduced in 1.51.0 ([#7431]), due to [its performance impact][#8065]. ([#8100])

[#7431]: tokio-rs/tokio#7431
[#8065]: tokio-rs/tokio#8065
[#8100]: tokio-rs/tokio#8100

Changes in 1.51.1
-----------------

1.51.1 (April 8th, 2026)
------------------------

Fixed
-----

- sync: fix semaphore reopens after forget ([#8021])
- net: surface errors from `SO_ERROR` on `recv` for UDP sockets on Linux ([#8001])

Fixed (unstable)
----------------

- metrics: fix `worker_local_schedule_count` test ([#8008])
- rt: do not leak fd when cancelling io\_uring open operation ([#7983])

[#7983]: tokio-rs/tokio#7983
[#8001]: tokio-rs/tokio#8001
[#8008]: tokio-rs/tokio#8008
[#8021]: tokio-rs/tokio#8021

Changes in 1.51.0
-----------------

1.51.0 (April 3rd, 2026)
------------------------

Added
-----

- net: implement `get_peer_cred` on Hurd ([#7989])
- runtime: add `tokio::runtime::worker_index()` ([#7921])
- runtime: add runtime name ([#7924])
- runtime: stabilize `LocalRuntime` ([#7557])
- wasm: add wasm32-wasip2 networking support ([#7933])

Changed
-------

- runtime: steal tasks from the LIFO slot ([#7431])

Fixed
-----

- docs: do not show "Available on non-loom only." doc label ([#7977])
- macros: improve overall macro hygiene ([#7997])
- sync: fix `notify_waiters` priority in `Notify` ([#7996])
- sync: fix panic in `Chan::recv_many` when called with non-empty vector on closed channel ([#7991])

[#7431]: tokio-rs/tokio#7431
[#7557]: tokio-rs/tokio#7557
[#7921]: tokio-rs/tokio#7921
[#7924]: tokio-rs/tokio#7924
[#7933]: tokio-rs/tokio#7933
[#7977]: tokio-rs/tokio#7977
[#7989]: tokio-rs/tokio#7989
[#7991]: tokio-rs/tokio#7991
[#7996]: tokio-rs/tokio#7996
[#7997]: tokio-rs/tokio#7997

Changes in 1.50.0
-----------------

1.50.0 (Mar 3rd, 2026)
----------------------

Added
-----

- net: add `TcpStream::set_zero_linger` ([#7837])
- rt: add `is_rt_shutdown_err` ([#7771])

Changed
-------

- io: add optimizer hint that `memchr` returns in-bounds pointer ([#7792])
- io: implement vectored writes for `write_buf` ([#7871])
- runtime: panic when `event_interval` is set to 0 ([#7838])
- runtime: shorten default thread name to fit in Linux limit ([#7880])
- signal: remember the result of `SetConsoleCtrlHandler` ([#7833])
- signal: specialize windows `Registry` ([#7885])

Fixed
-----

- io: always cleanup `AsyncFd` registration list on deregister ([#7773])
- macros: remove (most) local `use` declarations in `tokio::select!` ([#7929])
- net: fix `GET_BUF_SIZE` constant for `target_os = "android"` ([#7889])
- runtime: avoid redundant unpark in current_thread scheduler ([#7834])
- runtime: don't park in `current_thread` if `before_park` defers waker ([#7835])
- io: fix write readiness on ESP32 on short writes ([#7872])
- runtime: wake deferred tasks before entering `block_in_place` ([#7879])
- sync: drop rx waker when oneshot receiver is dropped ([#7886])
- runtime: fix double increment of `num_idle_threads` on shutdown ([#7910], [#7918], [#7922])

Unstable
--------

- fs: check for io-uring opcode support ([#7815])
- runtime: avoid lock acquisition after uring init ([#7850])

Documented
----------

- docs: update outdated unstable features section ([#7839])
- io: clarify the behavior of `AsyncWriteExt::shutdown()` ([#7908])
- io: explain how to flush stdout/stderr ([#7904])
- io: fix incorrect and confusing `AsyncWrite` documentation ([#7875])
- rt: clarify the documentation of `Runtime::spawn` ([#7803])
- rt: fix missing quotation in docs ([#7925])
- runtime: correct the default thread name in docs ([#7896])
- runtime: fix `event_interval` doc ([#7932])
- sync: clarify RwLock fairness documentation ([#7919])
- sync: clarify that `recv` returns `None` once closed and no more messages ([#7920])
- task: clarify when to use `spawn_blocking` vs dedicated threads ([#7923])
- task: doc that task drops before `JoinHandle` completion ([#7825])
- signal: guarantee that listeners never return `None` ([#7869])
- task: fix task module feature flags in docs ([#7891])
- task: fix two typos ([#7913])
- task: improve the docs of `Builder::spawn_local` ([#7828])
- time: add docs about auto-advance and when to use sleep ([#7858])
- util: fix typo in docs ([#7926])

[#7771]: tokio-rs/tokio#7771
[#7773]: tokio-rs/tokio#7773
[#7792]: tokio-rs/tokio#7792
[#7803]: tokio-rs/tokio#7803
[#7815]: tokio-rs/tokio#7815
[#7825]: tokio-rs/tokio#7825
[#7828]: tokio-rs/tokio#7828
[#7833]: tokio-rs/tokio#7833
[#7834]: tokio-rs/tokio#7834
[#7835]: tokio-rs/tokio#7835
[#7837]: tokio-rs/tokio#7837
[#7838]: tokio-rs/tokio#7838
[#7839]: tokio-rs/tokio#7839
[#7850]: tokio-rs/tokio#7850
[#7858]: tokio-rs/tokio#7858
[#7869]: tokio-rs/tokio#7869
[#7871]: tokio-rs/tokio#7871
[#7872]: tokio-rs/tokio#7872
[#7875]: tokio-rs/tokio#7875
[#7879]: tokio-rs/tokio#7879
[#7880]: tokio-rs/tokio#7880
[#7885]: tokio-rs/tokio#7885
[#7886]: tokio-rs/tokio#7886
[#7889]: tokio-rs/tokio#7889
[#7891]: tokio-rs/tokio#7891
[#7896]: tokio-rs/tokio#7896
[#7904]: tokio-rs/tokio#7904
[#7908]: tokio-rs/tokio#7908
[#7910]: tokio-rs/tokio#7910
[#7913]: tokio-rs/tokio#7913
[#7918]: tokio-rs/tokio#7918
[#7919]: tokio-rs/tokio#7919
[#7920]: tokio-rs/tokio#7920
[#7922]: tokio-rs/tokio#7922
[#7923]: tokio-rs/tokio#7923
[#7925]: tokio-rs/tokio#7925
[#7926]: tokio-rs/tokio#7926
[#7929]: tokio-rs/tokio#7929
[#7932]: tokio-rs/tokio#7932

Release notes:
- 1.53.1: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.53.1
- 1.53.0: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.53.0
- 1.52.4: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.4
- 1.52.3: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.3
- 1.52.2: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.2
- 1.52.1: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.1
- 1.52.0: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.52.0
- 1.51.4: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.4
- 1.51.3: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.3
- 1.51.2: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.2
- 1.51.1: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.1
- 1.51.0: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.51.0
- 1.50.0: https://github.com/tokio-rs/tokio/releases/tag/tokio-1.50.0

---

Transitive Dependencies
-----------------------

- libc (0.2.180 -> 0.2.189)
- mio (1.1.1 -> 1.2.3)
- socket2 (0.6.2 -> 0.6.5)
- syn 3.0.6 (added)
- tokio-macros (2.6.0 -> 2.7.2)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-tokio Area: The main tokio crate C-enhancement Category: A PR with an enhancement or bugfix. M-runtime Module: tokio/runtime R-loom-current-thread Run loom current-thread tests on this PR R-loom-multi-thread Run loom multi-thread tests on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rt: make the LIFO slot in the multi-threaded scheduler stealable

5 participants