Skip to content

fix(mtp): attribute error/timed-out/cancelled states as failures - #3560

Merged
richardwerkman merged 3 commits into
stryker-mutator:masterfrom
rlasjunies:fix/mtp-tunit-attribution
May 8, 2026
Merged

richardwerkman merged 3 commits into
stryker-mutator:masterfrom
rlasjunies:fix/mtp-tunit-attribution

Conversation

@rlasjunies

@rlasjunies rlasjunies commented Apr 24, 2026 •

Copy link
Copy Markdown
Contributor

closes #3570

Summary

The Microsoft Testing Platform (MTP) adapter only treated the "failed"
execution-state as a test failure, silently dropping tests that ended in
"error" (non-assertion exceptions), "timed-out" (per-test timeouts), or
"cancelled". Mutants that should have been Killed were reported as
Survived.

This is most visible when using TUnit + NSubstitute: Received(N)
assertions throw ReceivedCallsException, which TUnit — not recognising
it as a TUnit assertion — routes through its ErrorTestNode path and
serialises as "error" on the wire. The adapter's single-string filter
missed it.

What changed

  • New TestNodeStates helper holds the seven canonical MTP state strings
    (passed, skipped, failed, error, timed-out, cancelled,
    in-progress, plus discovered) and three classifiers (IsFinished,
    IsFailure, IsTimeout). One source of truth for every state literal.
  • SingleMicrosoftTestPlatformRunner result-building logic is extracted
    into an internal BuildTestRunResult method using the classifiers.
  • TimedOutTests is now populated from per-test timed-out states
    (previously always empty), so Mutant.AnalyzeTestRun can correctly
    classify mutants as Timeout rather than Survived.

Classification mapping:

State Bucket Resulting MutantStatus
failed / error / cancelled FailingTests Killed
timed-out TimedOutTests Timeout
passed / skipped executed, neither Survived (if covered)
in-progress / discovered excluded from executed n/a

Tests

29 new tests across TestNodeStatesTests.cs and
SingleMicrosoftTestPlatformRunnerTests.cs. Reverting the classifier to
its original buggy form makes 8 of them fail, proving regression coverage.
Full MTP adapter suite: 185 passed, 0 failed, 0 skipped.

Real-world impact

Apples-to-apples scoped run on a downstream TUnit + NSubstitute project
(same mutation set, same tests, only the adapter changed):

Baseline Fixed Delta
Killed 95 118 +23
Survived 126 103 -23
Strict score 42.99 % 53.39 % +10.40 pp

No drift in CompileError, Ignored or NoCoverage — exactly what an
adapter-only fix should look like.

Refs #3094

Test plan

  • dotnet test src/Stryker.slnx — 1969 passed, 0 failed, 1 pre-existing skip
  • dotnet test src/Stryker.TestRunner.MicrosoftTestPlatform.UnitTest — 185 passed, 0 failed
  • Temporarily reverted the classifier to reproduce the original bug — 8 new tests fail
  • End-to-end verification on a downstream TUnit + NSubstitute project — +23 kills, no regressions

🤖 Generated with Claude Code

rlasjunies and others added 2 commits April 24, 2026 22:46
The MTP adapter only treated "failed" as a failure, silently dropping tests
that ended in "error" (non-assertion exceptions — e.g. NSubstitute's
ReceivedCallsException routed through TUnit's ErrorTestNode), "timed-out"
(per-test timeouts) or "cancelled". Mutants that should have been Killed
were reported as Survived.

Classification is now centralised in the new internal TestNodeStates helper,
and TimedOutTests is populated from per-test timeout states so the mutation
engine can mark mutants as Timeout rather than collapsing everything into
Killed/Survived.

Full classification mapping:
 - failed / error / cancelled -> FailingTests (mutant Killed)
 - timed-out                  -> TimedOutTests (mutant Timeout)
 - passed / skipped           -> executed, neither failing nor timed-out
 - in-progress / discovered   -> excluded from executed tests

29 new unit tests cover every state. Reverting just the classifier to the
original buggy form made 8 of those tests fail, proving regression coverage.

Apples-to-apples verification on a downstream TUnit + NSubstitute project:
baseline 42.99% -> fixed 53.39% strict score on the same mutant set (23
mutants moved from Survived to Killed, no drift elsewhere).

Refs stryker-mutator#3094

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes false-negative failure attribution in the Microsoft Testing Platform (MTP) adapter by treating additional MTP execution-state values (error, timed-out, cancelled) as failure/timeout outcomes instead of only failed, preventing mutants from being incorrectly reported as Survived.

Changes:

  • Introduces TestNodeStates as a single source of truth for MTP execution-state literals and classification helpers (IsFinished, IsFailure, IsTimeout).
  • Refactors SingleMicrosoftTestPlatformRunner result mapping into an internal BuildTestRunResult method and correctly populates TimedOutTests based on per-test timed-out states.
  • Adds targeted unit tests covering state classification and result-bucketing regression scenarios.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/Stryker.TestRunner.MicrosoftTestPlatform/TestNodeStates.cs Adds canonical MTP state constants and classification helpers to avoid string-literal drift/bugs.
src/Stryker.TestRunner.MicrosoftTestPlatform/SingleMicrosoftTestPlatformRunner.cs Extracts and updates result-building to attribute error/cancelled as failures and timed-out as timeouts.
src/Stryker.TestRunner.MicrosoftTestPlatform/AssemblyTestServer.cs Replaces a hardcoded "discovered" state check with TestNodeStates.Discovered.
src/Stryker.TestRunner.MicrosoftTestPlatform.UnitTest/TestNodeStatesTests.cs Adds unit tests validating state constants and classifier behavior.
src/Stryker.TestRunner.MicrosoftTestPlatform.UnitTest/SingleMicrosoftTestPlatformRunnerTests.cs Adds regression tests ensuring state-to-bucket mapping produces correct failing/timed-out/executed test sets.

@richardwerkman

Copy link
Copy Markdown
Member

I was able to reproduce the issue and with this fix applied it was resolved. Thanks for your contribution!

@richardwerkman
richardwerkman enabled auto-merge (squash) May 8, 2026 13:08
@richardwerkman
richardwerkman merged commit 58969ca into stryker-mutator:master May 8, 2026
39 checks passed
tknightnd pushed a commit to NewDayTechnology/benchmarkdotnet.analyser that referenced this pull request May 20, 2026
Updated
[coverlet.collector](https://github.com/coverlet-coverage/coverlet) from
10.0.0 to 10.0.1.

<details>
<summary>Release notes</summary>

_Sourced from [coverlet.collector's
releases](https://github.com/coverlet-coverage/coverlet/releases)._

## 10.0.1

### Improvements

- Coverlet with MTP 2 doesn't show test coverage statistic in console
[#​1907](coverlet-coverage/coverlet#1907)
- Avoid unnecessary testhost restarts
[#​1912](coverlet-coverage/coverlet#1912) by
<https://github.com/mawosoft>

### Fixed

- Fix inconsistent paths in cobertura reports
[#​1723](coverlet-coverage/coverlet#1723)
- Fix when using "is" with "and" in pattern matching, branch coverage is
lower than normal
[#​1313](coverlet-coverage/coverlet#1313)
- Fix Coverlet flagging a branch for an async functions finally block
where none exists
[#​1337](coverlet-coverage/coverlet#1337)
- Fix Coverlet Tracker Missing CompilerGeneratedAttribute
[#​1828](coverlet-coverage/coverlet#1828)

### Maintenance

- Add architecture docs and diagrams for all integrations
[#​1927](coverlet-coverage/coverlet#1927)
- Update NuGet packages and .NET SDK versions
[#​1933](coverlet-coverage/coverlet#1933)

[Diff between 10.0.0 and
10.0.1](coverlet-coverage/coverlet@v10.0.0...v10.0.1)

Commits viewable in [compare
view](coverlet-coverage/coverlet@v10.0.0...v10.0.1).
</details>

Updated
[coverlet.msbuild](https://github.com/coverlet-coverage/coverlet) from
10.0.0 to 10.0.1.

<details>
<summary>Release notes</summary>

_Sourced from [coverlet.msbuild's
releases](https://github.com/coverlet-coverage/coverlet/releases)._

## 10.0.1

### Improvements

- Coverlet with MTP 2 doesn't show test coverage statistic in console
[#​1907](coverlet-coverage/coverlet#1907)
- Avoid unnecessary testhost restarts
[#​1912](coverlet-coverage/coverlet#1912) by
<https://github.com/mawosoft>

### Fixed

- Fix inconsistent paths in cobertura reports
[#​1723](coverlet-coverage/coverlet#1723)
- Fix when using "is" with "and" in pattern matching, branch coverage is
lower than normal
[#​1313](coverlet-coverage/coverlet#1313)
- Fix Coverlet flagging a branch for an async functions finally block
where none exists
[#​1337](coverlet-coverage/coverlet#1337)
- Fix Coverlet Tracker Missing CompilerGeneratedAttribute
[#​1828](coverlet-coverage/coverlet#1828)

### Maintenance

- Add architecture docs and diagrams for all integrations
[#​1927](coverlet-coverage/coverlet#1927)
- Update NuGet packages and .NET SDK versions
[#​1933](coverlet-coverage/coverlet#1933)

[Diff between 10.0.0 and
10.0.1](coverlet-coverage/coverlet@v10.0.0...v10.0.1)

Commits viewable in [compare
view](coverlet-coverage/coverlet@v10.0.0...v10.0.1).
</details>

Updated [dotnet-stryker](https://github.com/stryker-mutator/stryker-net)
from 4.14.1 to 4.14.2.

<details>
<summary>Release notes</summary>

_Sourced from [dotnet-stryker's
releases](https://github.com/stryker-mutator/stryker-net/releases)._

## 4.14.2

##
[4.14.2](https://github.com/stryker-mutator/stryker-net/compare/dotnet-stryker@​4.14.1...dotnet-stryker@​4.14.2)
(2026-05-17)


### Bug Fixes

* **mtp:** attribute error/timed-out/cancelled states as failures
([#​3560](stryker-mutator/stryker-net#3560))
([58969ca](stryker-mutator/stryker-net@58969ca)),
closes
[#​3094](stryker-mutator/stryker-net#3094)
* **mtp:** Disable DiffEngine and the Verify clipboard feature
([#​3547](stryker-mutator/stryker-net#3547))
([f7b2f20](stryker-mutator/stryker-net@f7b2f20))
* **mutating:** out var in lambdas triggering NRE
([#​3575](stryker-mutator/stryker-net#3575))
([5ce07cc](stryker-mutator/stryker-net@5ce07cc))
* **rollback:** Improve rollback for CS0165 and CS0177 compile errors
([#​3548](stryker-mutator/stryker-net#3548))
([b42be0e](stryker-mutator/stryker-net@b42be0e))


Commits viewable in [compare
view](https://github.com/stryker-mutator/stryker-net/compare/dotnet-stryker@​4.14.1...dotnet-stryker@​4.14.2).
</details>

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

MTP treats non assertion exceptions as passing.

4 participants