Skip to content

[xcode27.0] [devops] Recognize the "skipped, incorrect beta version" test summary - #26023

Merged
rolfbjarne merged 1 commit into
xcode27.0from
dev/rolf/fix-beta-skip-test-summary
Jul 10, 2026
Merged

rolfbjarne merged 1 commit into
xcode27.0from
dev/rolf/fix-beta-skip-test-summary

Conversation

@rolfbjarne

Copy link
Copy Markdown
Member

PR #26003 made run-packaged-macos-tests write a TestSummary.md whose first line is # ⚠️ <title>: Tests skipped, incorrect beta version when it skips tests because the current OS is a beta build that doesn't match the expected build version.

The CI summary parser (tools/devops/automation/scripts/TestResults.psm1) only knew about the "all passed", "no tests selected" and "# Test results" formats, so GetPassedTests threw Unable to understand the test result and the PR got :fire: Failed to compute test summaries comments instead of a proper summary.

This PR:

  • Recognizes the skip line in GetPassedTests (treating it as zero passed tests, tracked via a new hidden $Skipped flag).
  • Renders a proper :warning: ...: Tests skipped, incorrect beta version. message in the aggregated comment instead of the misleading "No tests selected.".
  • Adds a Pester regression test for the skip case.

🤖 Pull request created by Copilot

PR #26003 made run-packaged-macos-tests write a TestSummary.md whose first
line is "# ⚠️ <title>: Tests skipped, incorrect beta version" when it skips
tests because the current OS is a beta build that doesn't match the expected
build version.

The CI summary parser (TestResults.psm1) only knew about the "all passed",
"no tests selected" and "# Test results" formats, so it threw "Unable to
understand the test result" and posted a ":fire: Failed to compute test
summaries" comment on the PR.

Recognize the skip line (treating it as zero passed tests) and render a
proper ":warning: ...: Tests skipped, incorrect beta version." message in
the aggregated comment instead of the misleading "No tests selected."

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 9, 2026 17:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the Azure DevOps test-summary parser to correctly interpret the new “Tests skipped, incorrect beta version” TestSummary.md format generated by run-packaged-macos-tests, so CI can produce a meaningful aggregated summary instead of failing to compute one.

Changes:

  • Teach TestResult.GetPassedTests to recognize the beta-version-mismatch skip header and record a new $Skipped state.
  • Update aggregated success messaging to emit a dedicated warning line for skipped runs (instead of “No tests selected.”).
  • Add a Pester regression test covering the skipped-summary format.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
tools/devops/automation/scripts/TestResults.psm1 Parses the new skipped-summary header and propagates a “skipped” state into aggregated output.
tools/devops/automation/scripts/TestResults.Tests.ps1 Adds a regression test ensuring skipped summaries don’t throw and report zero passed tests.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

Copy link
Copy Markdown
Collaborator

✅ API diff for current PR / commit

NET (empty diffs)

✅ API diff vs stable

NET (empty diffs)

ℹ️ Generator diff

Generator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes)

Pipeline on Agent
Hash: a20222fc8fe01761cd33f9960eb7f2543e481814 [PR build]

@vs-mobiletools-engineering-service2

Copy link
Copy Markdown
Collaborator

🔥 [CI Build #a20222f] Test results 🔥

Test results

❌ Tests failed on VSTS: test results

1 tests crashed, 2 tests failed, 188 tests passed.

Failures

❌ dotnettests tests (iOS)

🔥 Failed catastrophically on VSTS: test results - dotnettests_ios (no summary found).

Html Report (VSDrops) Download

❌ dotnettests tests (macOS)

1 tests failed, 0 tests passed.

Failed tests

  • DotNet tests: Failed (Execution failed with exit code 1)
    • Xamarin.Tests.AppSizeTest.CoreCLR_Interpreter_TrimmableStatic(Ma...: App size changed significantly (+12,288 bytes (12.0 KB = 0.0 MB) different > tolerance of +-10,240 bytes (10.0 KB = 0.0 MB)). Ex...
    • Xamarin.Tests.AppSizeTest.CoreCLR_Interpreter(MacOSX,"osx-arm64;...: App size changed significantly (+13,312 bytes (13.0 KB = 0.0 MB) different > tolerance of +-10,240 bytes (10.0 KB = 0.0 MB)). Ex...

Html Report (VSDrops) Download

❌ monotouch tests (iOS)

1 tests failed, 17 tests passed.

Failed tests

  • monotouch-test/iOS - simulator/Debug (LinkSdk): TimedOut

Html Report (VSDrops) Download

Successes

✅ assembly-processing: All 1 tests passed. Html Report (VSDrops) Download
✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (tvOS): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 2 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 4 tests passed. Html Report (VSDrops) Download
✅ generator: All 5 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 4 tests passed. Html Report (VSDrops) Download
✅ introspection: All 6 tests passed. Html Report (VSDrops) Download
✅ linker (iOS): All 15 tests passed. Html Report (VSDrops) Download
✅ linker (MacCatalyst): All 15 tests passed. Html Report (VSDrops) Download
✅ linker (macOS): All 21 tests passed. Html Report (VSDrops) Download
✅ linker (tvOS): All 15 tests passed. Html Report (VSDrops) Download
✅ monotouch (MacCatalyst): All 17 tests passed. Html Report (VSDrops) Download
✅ monotouch (macOS): All 18 tests passed. Html Report (VSDrops) Download
✅ monotouch (tvOS): All 18 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ sharpie: All 1 tests passed. Html Report (VSDrops) Download
✅ windows: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. Html Report (VSDrops) Download
✅ xtro: All 1 tests passed. Html Report (VSDrops) Download

macOS tests

✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sequoia (15): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Tahoe (26): All 5 tests passed. Html Report (VSDrops) Download
⚠️ Tests on macOS Golden Gate (27): Tests skipped, incorrect beta version. Html Report (VSDrops) Download

Linux Build Verification

✅ Linux build succeeded

Pipeline on Agent
Hash: a20222fc8fe01761cd33f9960eb7f2543e481814 [PR build]

@rolfbjarne

Copy link
Copy Markdown
Member Author

Merging; test failures are unrelated.

@rolfbjarne
rolfbjarne merged commit aa64e9f into xcode27.0 Jul 10, 2026
53 of 56 checks passed
@rolfbjarne
rolfbjarne deleted the dev/rolf/fix-beta-skip-test-summary branch July 10, 2026 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants