Skip to content

regression in spawn_blocking caused by #7757 causes tests to hang #8056

Description

@hawkw

Version
tokio v1.52.0 (and specifically, commit 1604bc3)

`cargo tree | grep tokio` output (long)
eliza@hekate ~/Code/oxide/omicron $ cargo tree -p omicron-sled-agent | grep tokio
│   │   │   │   └── tokio v1.52.0
│   │   │   │       └── tokio-macros v2.7.0 (proc-macro)
│   │   │   │   └── tokio v1.52.0 (*)
│   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   │   │   ├── tokio-util v0.7.18
│   │   │   │   │   │   └── tokio v1.52.0 (*)
│   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   │   ├── tokio-rustls v0.26.4
│   │   │   │   │   └── tokio v1.52.0 (*)
│   │   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   │   │   ├── tokio-native-tls v0.3.1
│   │   │   │   │   │   └── tokio v1.52.0 (*)
│   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   │   ├── tokio-native-tls v0.3.1 (*)
│   │   │   │   ├── tokio-rustls v0.26.4 (*)
│   │   │   │   ├── tokio-util v0.7.18 (*)
│   │   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   │   ├── tokio-rustls v0.26.4 (*)
│   │   │   │   ├── tokio-util v0.7.18 (*)
│   │   │   ├── tokio v1.52.0 (*)
│   │   │   ├── tokio-postgres v0.7.16
│   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   │   ├── tokio-util v0.7.18 (*)
│   │   │   ├── tokio-rustls v0.26.4 (*)
│   │   │   ├── tokio-stream v0.1.18
│   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   │   └── tokio-util v0.7.18 (*)
│   │   │   ├── tokio-util v0.7.18 (*)
│   │   │   ├── tokio v1.52.0 (*)
│   │   │   ├── tokio-postgres v0.7.16 (*)
│   │   │   ├── tokio-rustls v0.26.4 (*)
│   │   │   ├── tokio-stream v0.1.18 (*)
│   │   │   ├── tokio-util v0.7.18 (*)
│   │   └── tokio v1.52.0 (*)
│   ├── tokio v1.52.0 (*)
│   │   ├── tokio v1.52.0 (*)
│   │   ├── tokio-rustls v0.25.0
│   │   │   └── tokio v1.52.0 (*)
│   │   │   └── tokio v1.52.0 (*)
│   │   │   └── tokio v1.52.0 (*)
│   │   ├── tokio v1.52.0 (*)
│   └── tokio v1.52.0 (*)
│   │   └── tokio v1.52.0 (*)
│   ├── tokio v1.52.0 (*)
│   │   ├── tokio v1.52.0 (*)
│   ├── tokio v1.52.0 (*)
│   ├── tokio v1.52.0 (*)
│   ├── tokio v1.52.0 (*)
│       ├── tokio v1.52.0 (*)
│   ├── tokio v1.52.0 (*)
│       │   │   └── tokio v1.52.0 (*)
│       │   └── tokio v1.52.0 (*)
│       ├── tokio v1.52.0 (*)
│   │   ├── tokio v1.52.0 (*)
│   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   ├── tokio v1.52.0 (*)
│   │   ├── tokio v1.52.0 (*)
│   │   ├── tokio-stream v0.1.18 (*)
│   ├── tokio v1.52.0 (*)
│   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   │   ├── tokio-util v0.7.18 (*)
│   │   │   │   └── tokio v1.52.0 (*)
│   │   │   ├── oxide-tokio-rt v0.1.3
│   │   │   │   └── tokio v1.52.0 (*)
│   │   │   ├── tokio v1.52.0 (*)
│   │   │   ├── tokio-util v0.7.18 (*)
│   │   │   ├── tokio v1.52.0 (*)
│   │   ├── tokio v1.52.0 (*)
│   ├── tokio-postgres v0.7.16 (*)
│   ├── tokio v1.52.0 (*)
├── oxide-tokio-rt v0.1.3 (*)
│   ├── tokio v1.52.0 (*)
│   ├── tokio v1.52.0 (*)
│   ├── tokio v1.52.0 (*)
│   ├── tokio-tungstenite v0.21.0
│   │   ├── tokio v1.52.0 (*)
│   │   │   └── tokio v1.52.0 (*)
│   │   ├── tokio v1.52.0 (*)
│   │   ├── tokio-rustls v0.25.0 (*)
│   │   ├── tokio-util v0.7.18 (*)
│   ├── tokio v1.52.0 (*)
│   ├── tokio-tungstenite v0.21.0 (*)
│   │   │   └── tokio v1.52.0 (*)
│   │   ├── tokio v1.52.0 (*)
│   │   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   │   ├── tokio v1.52.0 (*)
│   │   │   │   ├── tokio-stream v0.1.18 (*)
│   │   │   └── tokio v1.52.0 (*)
│   ├── tokio v1.52.0 (*)
│   └── tokio v1.52.0 (*)
│   └── tokio v1.52.0 (*)
│   ├── oxide-tokio-rt v0.1.3 (*)
│   ├── tokio v1.52.0 (*)
│   ├── tokio v1.52.0 (*)
│   ├── tokio-rustls v0.26.4 (*)
├── tokio v1.52.0 (*)
├── tokio-stream v0.1.18 (*)
├── tokio-util v0.7.18 (*)
│   │   │   ├── tokio v1.52.0 (*)
│   │   │   ├── tokio-util v0.7.18 (*)
│   │   ├── oxide-tokio-rt v0.1.3 (*)
│   │   ├── tokio v1.52.0 (*)
│   ├── tokio v1.52.0 (*)
│   ├── oxide-tokio-rt v0.1.3 (*)
│   ├── tokio v1.52.0 (*)
│   ├── tokio-postgres v0.7.16 (*)
├── tokio-stream v0.1.18 (*)
├── tokio-util v0.7.18 (*)

Platform

eliza@hekate ~/Code/oxide/omicron $ uname -a
Linux hekate 6.12.55 #1-NixOS SMP PREEMPT_DYNAMIC Thu Oct 23 14:20:48 UTC 2025 x86_64 GNU/Linux

Description

After updating our application (oxidecomputer/omicron) to depend on Tokio v1.52.0, we noticed that some of our tests, which exercises code that makes use of tokio::spawn_blocking to perform file I/O, will now sometimes hang indefinitely. This is described in oxidecomputer/omicron#10272.

Since we were updating from Tokio v1.50.0 to v1.52.0, I bisected which Tokio versions introduced the hang, and determined that the regression was introduced in oxidecomputer/omicron#10277 (comment). Knowing that our code was using spawn_blocking, and #7757 changed the internal implementation of spawn_blocking somewhat substantially, I tried patching our Tokio dependency to specific Tokio commits. I found that on the commit immediately prior to the one that merged #7757, commit 4c7d8b2, the hang does not reproduce, but that on 1604bc3 (the commit that merged #7757) does reproduce the hang. See oxidecomputer/omicron#10277 (comment) for details. So, I can say conclusively that 1604bc3 has caused at least some previously-fine programs to hang indefinitely.

I don't have a minimum reproducer yet, but will try to put one together. In the meantime, this is the test in which we noticed the hang: https://github.com/oxidecomputer/omicron/blob/0ef430324bbcc83b8fe3c733ce7629f2fd4faf93/sled-agent/src/artifact_store.rs#L1228-L1313

Activity

  1. added
    C-bugCategory: This is a bug.
    A-tokioArea: The main tokio crate
    M-runtimeModule: tokio/runtime
    I-hangProgram never terminates, resulting from infinite loops, deadlock, livelock, etc.
    M-blockingModule: tokio/task/blocking
    on Apr 16, 2026
  2. added theissue type on Apr 16, 2026
  3. hawkw commented on Apr 16, 2026

    @hawkw
    MemberAuthor

    Personally, I would really like to get a v1.52.1 out that either moves #7757's sharded blocking inject to an unstable opt-in runtime::Builder setting for now, or reverts the change entirely, as I'd really like to be able to pick up #8010 without having the regression...

  4. added 2 commits that reference this issue on Apr 16, 2026
  5. hawkw commented on Apr 16, 2026

    @hawkw
    MemberAuthor

    I had hoped that it would be possible to move the sharded blocking queue from #7757 into an opt-in unstable tokio::runtime::Builder feature. However, unfortunately, this isn't super easy to put behind an opt-in toggle, as the sharded queue also requires changes to the code in spawn_blocking itself, making it harder to switch between the sharded queue and the previous Mutex<VecDeque> implementation. I had hoped we could just make the value of NUM_SHARDS , here:
    https://github.com/alex/tokio/blob/9de7950e59f8acea412600c2102ab592c419483a/tokio/src/runtime/blocking/sharded_queue.rs#L29-L32
    into a variable, which gets set to either 1 or 16 based on the builder parameter. Sadly, testing this locally by patching Tokio to just always set NUM_SHARDS = 1, I can still reproduce the hang. Therefore, in the interest of just getting a fix back as soon as possible, I've opened #8057 to just revert #7757 for now, so that we can publish a patch release to fix the regression.

  6. 6 remaining items

  7. hawkw commented on Apr 17, 2026

    @hawkw
    MemberAuthor

    Ah, is this reproducibel entirely on the OSS test suite? If so, that's probably sufficient to run with.

    Our test, oxidecomputer/omicron#10277, reproduces this issue pretty consistently within two or three iterations. If you check out the linked issue, there should be a cargo nextest run command that will hang pretty consistently within the first couple of iterations. The linked issue should also have a commit you can check out with the regression already in the lockfile.

  8. alex commented on Apr 17, 2026

    @alex
    Contributor

    Perfect, thank you!

  9. philjb commented on Apr 20, 2026

    @philjb

    Thanks for fixing (reverting) this. I just stumbled into this ticket from the same similar sounding symptoms. We have a test that triggers a code path with spawn_blocking; the test had been stable but recently began to fail as tasks were delayed (looked like additional spawn_blocking calls were not starting new threads and tasks would run serially instead of concurrently). Traced through to tokio's code and got to this ticket. thanks again!

  10. added a commit that references this issue on Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-tokioArea: The main tokio crateC-bugCategory: This is a bug.I-hangProgram never terminates, resulting from infinite loops, deadlock, livelock, etc.M-blockingModule: tokio/task/blockingM-runtimeModule: tokio/runtime

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions