Repository navigation
fix(mtp): attribute error/timed-out/cancelled states as failures - #3560
Merged
richardwerkman merged 3 commits intoMay 8, 2026
Merged
Conversation
The MTP adapter only treated "failed" as a failure, silently dropping tests that ended in "error" (non-assertion exceptions — e.g. NSubstitute's ReceivedCallsException routed through TUnit's ErrorTestNode), "timed-out" (per-test timeouts) or "cancelled". Mutants that should have been Killed were reported as Survived. Classification is now centralised in the new internal TestNodeStates helper, and TimedOutTests is populated from per-test timeout states so the mutation engine can mark mutants as Timeout rather than collapsing everything into Killed/Survived. Full classification mapping: - failed / error / cancelled -> FailingTests (mutant Killed) - timed-out -> TimedOutTests (mutant Timeout) - passed / skipped -> executed, neither failing nor timed-out - in-progress / discovered -> excluded from executed tests 29 new unit tests cover every state. Reverting just the classifier to the original buggy form made 8 of those tests fail, proving regression coverage. Apples-to-apples verification on a downstream TUnit + NSubstitute project: baseline 42.99% -> fixed 53.39% strict score on the same mutant set (23 mutants moved from Survived to Killed, no drift elsewhere). Refs stryker-mutator#3094 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes false-negative failure attribution in the Microsoft Testing Platform (MTP) adapter by treating additional MTP execution-state values (error, timed-out, cancelled) as failure/timeout outcomes instead of only failed, preventing mutants from being incorrectly reported as Survived.
Changes:
- Introduces
TestNodeStatesas a single source of truth for MTP execution-state literals and classification helpers (IsFinished,IsFailure,IsTimeout). - Refactors
SingleMicrosoftTestPlatformRunnerresult mapping into aninternal BuildTestRunResultmethod and correctly populatesTimedOutTestsbased on per-testtimed-outstates. - Adds targeted unit tests covering state classification and result-bucketing regression scenarios.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| src/Stryker.TestRunner.MicrosoftTestPlatform/TestNodeStates.cs | Adds canonical MTP state constants and classification helpers to avoid string-literal drift/bugs. |
| src/Stryker.TestRunner.MicrosoftTestPlatform/SingleMicrosoftTestPlatformRunner.cs | Extracts and updates result-building to attribute error/cancelled as failures and timed-out as timeouts. |
| src/Stryker.TestRunner.MicrosoftTestPlatform/AssemblyTestServer.cs | Replaces a hardcoded "discovered" state check with TestNodeStates.Discovered. |
| src/Stryker.TestRunner.MicrosoftTestPlatform.UnitTest/TestNodeStatesTests.cs | Adds unit tests validating state constants and classifier behavior. |
| src/Stryker.TestRunner.MicrosoftTestPlatform.UnitTest/SingleMicrosoftTestPlatformRunnerTests.cs | Adds regression tests ensuring state-to-bucket mapping produces correct failing/timed-out/executed test sets. |
Member
|
I was able to reproduce the issue and with this fix applied it was resolved. Thanks for your contribution! |
richardwerkman
enabled auto-merge (squash)
May 8, 2026 13:08
This was referenced May 18, 2026
Merged
tknightnd
pushed a commit
to NewDayTechnology/benchmarkdotnet.analyser
that referenced
this pull request
May 20, 2026
Updated [coverlet.collector](https://github.com/coverlet-coverage/coverlet) from 10.0.0 to 10.0.1. <details> <summary>Release notes</summary> _Sourced from [coverlet.collector's releases](https://github.com/coverlet-coverage/coverlet/releases)._ ## 10.0.1 ### Improvements - Coverlet with MTP 2 doesn't show test coverage statistic in console [#1907](coverlet-coverage/coverlet#1907) - Avoid unnecessary testhost restarts [#1912](coverlet-coverage/coverlet#1912) by <https://github.com/mawosoft> ### Fixed - Fix inconsistent paths in cobertura reports [#1723](coverlet-coverage/coverlet#1723) - Fix when using "is" with "and" in pattern matching, branch coverage is lower than normal [#1313](coverlet-coverage/coverlet#1313) - Fix Coverlet flagging a branch for an async functions finally block where none exists [#1337](coverlet-coverage/coverlet#1337) - Fix Coverlet Tracker Missing CompilerGeneratedAttribute [#1828](coverlet-coverage/coverlet#1828) ### Maintenance - Add architecture docs and diagrams for all integrations [#1927](coverlet-coverage/coverlet#1927) - Update NuGet packages and .NET SDK versions [#1933](coverlet-coverage/coverlet#1933) [Diff between 10.0.0 and 10.0.1](coverlet-coverage/coverlet@v10.0.0...v10.0.1) Commits viewable in [compare view](coverlet-coverage/coverlet@v10.0.0...v10.0.1). </details> Updated [coverlet.msbuild](https://github.com/coverlet-coverage/coverlet) from 10.0.0 to 10.0.1. <details> <summary>Release notes</summary> _Sourced from [coverlet.msbuild's releases](https://github.com/coverlet-coverage/coverlet/releases)._ ## 10.0.1 ### Improvements - Coverlet with MTP 2 doesn't show test coverage statistic in console [#1907](coverlet-coverage/coverlet#1907) - Avoid unnecessary testhost restarts [#1912](coverlet-coverage/coverlet#1912) by <https://github.com/mawosoft> ### Fixed - Fix inconsistent paths in cobertura reports [#1723](coverlet-coverage/coverlet#1723) - Fix when using "is" with "and" in pattern matching, branch coverage is lower than normal [#1313](coverlet-coverage/coverlet#1313) - Fix Coverlet flagging a branch for an async functions finally block where none exists [#1337](coverlet-coverage/coverlet#1337) - Fix Coverlet Tracker Missing CompilerGeneratedAttribute [#1828](coverlet-coverage/coverlet#1828) ### Maintenance - Add architecture docs and diagrams for all integrations [#1927](coverlet-coverage/coverlet#1927) - Update NuGet packages and .NET SDK versions [#1933](coverlet-coverage/coverlet#1933) [Diff between 10.0.0 and 10.0.1](coverlet-coverage/coverlet@v10.0.0...v10.0.1) Commits viewable in [compare view](coverlet-coverage/coverlet@v10.0.0...v10.0.1). </details> Updated [dotnet-stryker](https://github.com/stryker-mutator/stryker-net) from 4.14.1 to 4.14.2. <details> <summary>Release notes</summary> _Sourced from [dotnet-stryker's releases](https://github.com/stryker-mutator/stryker-net/releases)._ ## 4.14.2 ## [4.14.2](https://github.com/stryker-mutator/stryker-net/compare/dotnet-stryker@4.14.1...dotnet-stryker@4.14.2) (2026-05-17) ### Bug Fixes * **mtp:** attribute error/timed-out/cancelled states as failures ([#3560](stryker-mutator/stryker-net#3560)) ([58969ca](stryker-mutator/stryker-net@58969ca)), closes [#3094](stryker-mutator/stryker-net#3094) * **mtp:** Disable DiffEngine and the Verify clipboard feature ([#3547](stryker-mutator/stryker-net#3547)) ([f7b2f20](stryker-mutator/stryker-net@f7b2f20)) * **mutating:** out var in lambdas triggering NRE ([#3575](stryker-mutator/stryker-net#3575)) ([5ce07cc](stryker-mutator/stryker-net@5ce07cc)) * **rollback:** Improve rollback for CS0165 and CS0177 compile errors ([#3548](stryker-mutator/stryker-net#3548)) ([b42be0e](stryker-mutator/stryker-net@b42be0e)) Commits viewable in [compare view](https://github.com/stryker-mutator/stryker-net/compare/dotnet-stryker@4.14.1...dotnet-stryker@4.14.2). </details> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced May 21, 2026
This was referenced Jul 20, 2026
This was referenced Sep 3, 2026
This was referenced Sep 14, 2026
This was referenced Sep 24, 2026
This was referenced Oct 29, 2026
This was referenced Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #3570
Summary
The Microsoft Testing Platform (MTP) adapter only treated the
"failed"execution-state as a test failure, silently dropping tests that ended in
"error"(non-assertion exceptions),"timed-out"(per-test timeouts), or"cancelled". Mutants that should have been Killed were reported asSurvived.
This is most visible when using TUnit + NSubstitute:
Received(N)assertions throw
ReceivedCallsException, which TUnit — not recognisingit as a TUnit assertion — routes through its
ErrorTestNodepath andserialises as
"error"on the wire. The adapter's single-string filtermissed it.
What changed
TestNodeStateshelper holds the seven canonical MTP state strings(
passed,skipped,failed,error,timed-out,cancelled,in-progress, plusdiscovered) and three classifiers (IsFinished,IsFailure,IsTimeout). One source of truth for every state literal.SingleMicrosoftTestPlatformRunnerresult-building logic is extractedinto an
internal BuildTestRunResultmethod using the classifiers.TimedOutTestsis now populated from per-testtimed-outstates(previously always empty), so
Mutant.AnalyzeTestRuncan correctlyclassify mutants as
Timeoutrather thanSurvived.Classification mapping:
MutantStatusfailed/error/cancelledFailingTeststimed-outTimedOutTestspassed/skippedin-progress/discoveredTests
29 new tests across
TestNodeStatesTests.csandSingleMicrosoftTestPlatformRunnerTests.cs. Reverting the classifier toits original buggy form makes 8 of them fail, proving regression coverage.
Full MTP adapter suite: 185 passed, 0 failed, 0 skipped.
Real-world impact
Apples-to-apples scoped run on a downstream TUnit + NSubstitute project
(same mutation set, same tests, only the adapter changed):
No drift in
CompileError,IgnoredorNoCoverage— exactly what anadapter-only fix should look like.
Refs #3094
Test plan
dotnet test src/Stryker.slnx— 1969 passed, 0 failed, 1 pre-existing skipdotnet test src/Stryker.TestRunner.MicrosoftTestPlatform.UnitTest— 185 passed, 0 failed🤖 Generated with Claude Code