Skip to content

Test sequence causes tests to consistently fail or pass #9499

Open

Description

@barbeque-squared

Describe the bug

I am experiencing flaky tests. I've tracked it down to it something having to do with mocks/spies in combination with two specific tests being executed in a specific order by 1 worker/thread/fork/thing.

We (ideally) don't use test isolation, but from what I understand that's only if you start messing with global stuff (eg fetch, process.env) anyway. I've also tried various options of clearMocks, mockReset and restoreMocks, and it seems to trigger in all three unless I enable test isolation.

I understand that disabling test isolation might cause tests to have side effects that leak over to other tests, but I've narrowed it down to a reproducable example where it's just two very simple test files which if ordered alphabetically consistently fail. And I don't understand why. It also feels like this behaviour has gotten more common since vitest 4.

it's possible that this isn't technically a bug, or maybe our setup is just wrong, but I can't find anything about this in the documentation nor existing issues. But explicitly mocking a file and it still somehow calling the real thing feels suspicious, regardless of the test order.

Reproduction

https://github.com/barbeque-squared/vitest-mockbug

npm ci
npm run test # will fail
npm run test:pass # will pass, the only difference is the test order

System Info

System:
    OS: Linux 6.18 Arch Linux
    CPU: (12) x64 13th Gen Intel(R) Core(TM) i5-1335U
    Memory: 5.59 GB / 15.28 GB
    Container: Yes
    Shell: 5.3.9 - /usr/bin/bash
  Binaries:
    Node: 25.3.0 - /usr/bin/node
    npm: 11.7.0 - /usr/bin/npm
    pnpm: 10.28.0 - /usr/bin/pnpm
  Browsers:
    Chromium: 144.0.7559.59
    Firefox: 147.0.1
    Firefox Developer Edition: 147.0.1
  npmPackages:
    vitest: ^4.0.17 => 4.0.17

Used Package Manager

npm

Validations

Activity

  1. sheremet-va commented on Jan 22, 2026

    @sheremet-va
    Member

    This happens because ./file2.ts was already executed and cached (since you disabled isolation), so the cached version is returned which had no module mock at the time.

    Invalidating the module automatically would just revert isolate: true to false, so we don't do it.

    You can also notice that ./ExternalPackageMessageValidator imported inside the test file itself is a mocked module returned from vi.mock factory because ./file2.test.ts wasn't executed yet, so it hits the "is module mocked" check and returns a mock, but that module references a different object than the one in ./file2.ts.

  2. sheremet-va commented on Jan 22, 2026

    @sheremet-va
    Member

    What vi.mock does is that it returns a different module when import is executed. Since file2 is not executed again, the module in that file is not actually mocked.

  3. sheremet-va commented on Jan 22, 2026

    @sheremet-va
    Member

    There is a way for Vitest to dynamically update that module, bun does that (it's not implemented by us), but I'm not sure if it's a good idea to implement, need to think on the pros and cons. One con is that top level expressions won't be revalidated:

    import { validate } from 'mocked-module'
    const valid = validate('123') // not updated
    export function validateDynamically(name) {
      return validate(name) // updated with a mock
    }
  4. barbeque-squared commented on Jan 22, 2026

    @barbeque-squared
    Author

    Thanks for this explanation. It's clearly intended behaviour then, and it's not too hard to rewrite this with vi.spyOn which I've more or less started to prefer (because it seems as long as you don't blindly .mockImplementation it Just Works and doesn't carry over -- .mockImplementationOnce is totally fine as long as people verify that it was actually called)

    Perhaps related: is it possible that vi.unmock does apply only to the 1 testfile it's called in, even in isolate: false ? Because from my observations that does look like what is happening. (I vi.mock some prisma helper from the setup file and only vi.unmock it in the one test for said helper -- and all other tests still use the mock from the setup file regardless of order).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions