Repository navigation
Description
Version
tokio v1.52.0 (and specifically, commit 1604bc3)
`cargo tree | grep tokio` output (long)
eliza@hekate ~/Code/oxide/omicron $ cargo tree -p omicron-sled-agent | grep tokio
│ │ │ │ └── tokio v1.52.0
│ │ │ │ └── tokio-macros v2.7.0 (proc-macro)
│ │ │ │ └── tokio v1.52.0 (*)
│ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ │ │ ├── tokio-util v0.7.18
│ │ │ │ │ │ └── tokio v1.52.0 (*)
│ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ │ ├── tokio-rustls v0.26.4
│ │ │ │ │ └── tokio v1.52.0 (*)
│ │ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ │ │ ├── tokio-native-tls v0.3.1
│ │ │ │ │ │ └── tokio v1.52.0 (*)
│ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ │ ├── tokio-native-tls v0.3.1 (*)
│ │ │ │ ├── tokio-rustls v0.26.4 (*)
│ │ │ │ ├── tokio-util v0.7.18 (*)
│ │ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ │ ├── tokio-rustls v0.26.4 (*)
│ │ │ │ ├── tokio-util v0.7.18 (*)
│ │ │ ├── tokio v1.52.0 (*)
│ │ │ ├── tokio-postgres v0.7.16
│ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ │ ├── tokio-util v0.7.18 (*)
│ │ │ ├── tokio-rustls v0.26.4 (*)
│ │ │ ├── tokio-stream v0.1.18
│ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ │ └── tokio-util v0.7.18 (*)
│ │ │ ├── tokio-util v0.7.18 (*)
│ │ │ ├── tokio v1.52.0 (*)
│ │ │ ├── tokio-postgres v0.7.16 (*)
│ │ │ ├── tokio-rustls v0.26.4 (*)
│ │ │ ├── tokio-stream v0.1.18 (*)
│ │ │ ├── tokio-util v0.7.18 (*)
│ │ └── tokio v1.52.0 (*)
│ ├── tokio v1.52.0 (*)
│ │ ├── tokio v1.52.0 (*)
│ │ ├── tokio-rustls v0.25.0
│ │ │ └── tokio v1.52.0 (*)
│ │ │ └── tokio v1.52.0 (*)
│ │ │ └── tokio v1.52.0 (*)
│ │ ├── tokio v1.52.0 (*)
│ └── tokio v1.52.0 (*)
│ │ └── tokio v1.52.0 (*)
│ ├── tokio v1.52.0 (*)
│ │ ├── tokio v1.52.0 (*)
│ ├── tokio v1.52.0 (*)
│ ├── tokio v1.52.0 (*)
│ ├── tokio v1.52.0 (*)
│ ├── tokio v1.52.0 (*)
│ ├── tokio v1.52.0 (*)
│ │ │ └── tokio v1.52.0 (*)
│ │ └── tokio v1.52.0 (*)
│ ├── tokio v1.52.0 (*)
│ │ ├── tokio v1.52.0 (*)
│ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ ├── tokio v1.52.0 (*)
│ │ ├── tokio v1.52.0 (*)
│ │ ├── tokio-stream v0.1.18 (*)
│ ├── tokio v1.52.0 (*)
│ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ │ ├── tokio-util v0.7.18 (*)
│ │ │ │ └── tokio v1.52.0 (*)
│ │ │ ├── oxide-tokio-rt v0.1.3
│ │ │ │ └── tokio v1.52.0 (*)
│ │ │ ├── tokio v1.52.0 (*)
│ │ │ ├── tokio-util v0.7.18 (*)
│ │ │ ├── tokio v1.52.0 (*)
│ │ ├── tokio v1.52.0 (*)
│ ├── tokio-postgres v0.7.16 (*)
│ ├── tokio v1.52.0 (*)
├── oxide-tokio-rt v0.1.3 (*)
│ ├── tokio v1.52.0 (*)
│ ├── tokio v1.52.0 (*)
│ ├── tokio v1.52.0 (*)
│ ├── tokio-tungstenite v0.21.0
│ │ ├── tokio v1.52.0 (*)
│ │ │ └── tokio v1.52.0 (*)
│ │ ├── tokio v1.52.0 (*)
│ │ ├── tokio-rustls v0.25.0 (*)
│ │ ├── tokio-util v0.7.18 (*)
│ ├── tokio v1.52.0 (*)
│ ├── tokio-tungstenite v0.21.0 (*)
│ │ │ └── tokio v1.52.0 (*)
│ │ ├── tokio v1.52.0 (*)
│ │ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ │ ├── tokio v1.52.0 (*)
│ │ │ │ ├── tokio-stream v0.1.18 (*)
│ │ │ └── tokio v1.52.0 (*)
│ ├── tokio v1.52.0 (*)
│ └── tokio v1.52.0 (*)
│ └── tokio v1.52.0 (*)
│ ├── oxide-tokio-rt v0.1.3 (*)
│ ├── tokio v1.52.0 (*)
│ ├── tokio v1.52.0 (*)
│ ├── tokio-rustls v0.26.4 (*)
├── tokio v1.52.0 (*)
├── tokio-stream v0.1.18 (*)
├── tokio-util v0.7.18 (*)
│ │ │ ├── tokio v1.52.0 (*)
│ │ │ ├── tokio-util v0.7.18 (*)
│ │ ├── oxide-tokio-rt v0.1.3 (*)
│ │ ├── tokio v1.52.0 (*)
│ ├── tokio v1.52.0 (*)
│ ├── oxide-tokio-rt v0.1.3 (*)
│ ├── tokio v1.52.0 (*)
│ ├── tokio-postgres v0.7.16 (*)
├── tokio-stream v0.1.18 (*)
├── tokio-util v0.7.18 (*)Platform
eliza@hekate ~/Code/oxide/omicron $ uname -a
Linux hekate 6.12.55 #1-NixOS SMP PREEMPT_DYNAMIC Thu Oct 23 14:20:48 UTC 2025 x86_64 GNU/LinuxDescription
After updating our application (oxidecomputer/omicron) to depend on Tokio v1.52.0, we noticed that some of our tests, which exercises code that makes use of tokio::spawn_blocking to perform file I/O, will now sometimes hang indefinitely. This is described in oxidecomputer/omicron#10272.
Since we were updating from Tokio v1.50.0 to v1.52.0, I bisected which Tokio versions introduced the hang, and determined that the regression was introduced in oxidecomputer/omicron#10277 (comment). Knowing that our code was using spawn_blocking, and #7757 changed the internal implementation of spawn_blocking somewhat substantially, I tried patching our Tokio dependency to specific Tokio commits. I found that on the commit immediately prior to the one that merged #7757, commit 4c7d8b2, the hang does not reproduce, but that on 1604bc3 (the commit that merged #7757) does reproduce the hang. See oxidecomputer/omicron#10277 (comment) for details. So, I can say conclusively that 1604bc3 has caused at least some previously-fine programs to hang indefinitely.
I don't have a minimum reproducer yet, but will try to put one together. In the meantime, this is the test in which we noticed the hang: https://github.com/oxidecomputer/omicron/blob/0ef430324bbcc83b8fe3c733ce7629f2fd4faf93/sled-agent/src/artifact_store.rs#L1228-L1313
Activity
revert(agent-data-plane): update to tokio 1.52 and enable eager drive…
revert(agent-data-plane): update to tokio 1.52 and enable eager drive…
hawkw commentedon Apr 16, 2026
Personally, I would really like to get a v1.52.1 out that either moves #7757's sharded blocking inject to an unstable opt-in
runtime::Buildersetting for now, or reverts the change entirely, as I'd really like to be able to pick up #8010 without having the regression...revert #10272 due to tokio-rs/tokio#8056 (#10279)
rt: revert #7757 to fix regression in `spawn_blocking`
hawkw commentedon Apr 16, 2026
I had hoped that it would be possible to move the sharded blocking queue from #7757 into an opt-in unstable
tokio::runtime::Builderfeature. However, unfortunately, this isn't super easy to put behind an opt-in toggle, as the sharded queue also requires changes to the code inspawn_blockingitself, making it harder to switch between the sharded queue and the previousMutex<VecDeque>implementation. I had hoped we could just make the value ofNUM_SHARDS, here:https://github.com/alex/tokio/blob/9de7950e59f8acea412600c2102ab592c419483a/tokio/src/runtime/blocking/sharded_queue.rs#L29-L32
into a variable, which gets set to either 1 or 16 based on the builder parameter. Sadly, testing this locally by patching Tokio to just always set
NUM_SHARDS = 1, I can still reproduce the hang. Therefore, in the interest of just getting a fix back as soon as possible, I've opened #8057 to just revert #7757 for now, so that we can publish a patch release to fix the regression.chore: prepare to release v1.52.1
chore: prepare to release v1.52.1 (#8059)
6 remaining items
hawkw commentedon Apr 17, 2026
Our test, oxidecomputer/omicron#10277, reproduces this issue pretty consistently within two or three iterations. If you check out the linked issue, there should be a
cargo nextest runcommand that will hang pretty consistently within the first couple of iterations. The linked issue should also have a commit you can check out with the regression already in the lockfile.alex commentedon Apr 17, 2026
Perfect, thank you!
chore(agent-data-plane): update to tokio 1.52.1 and re-enable eager d…
feat(deps): update rust crate tokio (1.51.1 → 1.52.1) (#11)
introduce a regression test for tokio-rs#8056
introduce a regression test for tokio-rs#8056
introduce a regression test for tokio-rs#8056
philjb commentedon Apr 20, 2026
Thanks for fixing (reverting) this. I just stumbled into this ticket from the same similar sounding symptoms. We have a test that triggers a code path with spawn_blocking; the test had been stable but recently began to fail as tasks were delayed (looked like additional spawn_blocking calls were not starting new threads and tasks would run serially instead of concurrently). Traced through to tokio's code and got to this ticket. thanks again!
blocking: introduce a regression test for #8056 (#8068)
revert(agent-data-plane): update to tokio 1.52 and enable eager drive…
runtime: add an opt-in sharded spawn_blocking queue
runtime: add an opt-in sharded spawn_blocking queue (#8337)
Update tokio (1.49.0 -> 1.53.1)
Update tokio to 1.53.3 to stop responses getting stuck
Update tokio to 1.53.3 to stop responses getting stuck