Repository navigation
[xcode27.0] [devops] Recognize the "skipped, incorrect beta version" test summary - #26023
Conversation
PR #26003 made run-packaged-macos-tests write a TestSummary.md whose first line is "#⚠️ <title>: Tests skipped, incorrect beta version" when it skips tests because the current OS is a beta build that doesn't match the expected build version. The CI summary parser (TestResults.psm1) only knew about the "all passed", "no tests selected" and "# Test results" formats, so it threw "Unable to understand the test result" and posted a ":fire: Failed to compute test summaries" comment on the PR. Recognize the skip line (treating it as zero passed tests) and render a proper ":warning: ...: Tests skipped, incorrect beta version." message in the aggregated comment instead of the misleading "No tests selected." Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Updates the Azure DevOps test-summary parser to correctly interpret the new “Tests skipped, incorrect beta version” TestSummary.md format generated by run-packaged-macos-tests, so CI can produce a meaningful aggregated summary instead of failing to compute one.
Changes:
- Teach
TestResult.GetPassedTeststo recognize the beta-version-mismatch skip header and record a new$Skippedstate. - Update aggregated success messaging to emit a dedicated warning line for skipped runs (instead of “No tests selected.”).
- Add a Pester regression test covering the skipped-summary format.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| tools/devops/automation/scripts/TestResults.psm1 | Parses the new skipped-summary header and propagates a “skipped” state into aggregated output. |
| tools/devops/automation/scripts/TestResults.Tests.ps1 | Adds a regression test ensuring skipped summaries don’t throw and report zero passed tests. |
This comment has been minimized.
This comment has been minimized.
✅ API diff for current PR / commitNET (empty diffs)✅ API diff vs stableNET (empty diffs)ℹ️ Generator diffGenerator Diff: vsdrops (html) vsdrops (raw diff) gist (raw diff) - Please review changes) Pipeline on Agent |
🔥 [CI Build #a20222f] Test results 🔥Test results❌ Tests failed on VSTS: test results 1 tests crashed, 2 tests failed, 188 tests passed. Failures❌ dotnettests tests (iOS)🔥 Failed catastrophically on VSTS: test results - dotnettests_ios (no summary found). Html Report (VSDrops) Download ❌ dotnettests tests (macOS)1 tests failed, 0 tests passed.Failed tests
Html Report (VSDrops) Download ❌ monotouch tests (iOS)1 tests failed, 17 tests passed.Failed tests
Html Report (VSDrops) Download Successes✅ assembly-processing: All 1 tests passed. Html Report (VSDrops) Download macOS tests✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download Linux Build VerificationPipeline on Agent |
|
Merging; test failures are unrelated. |
PR #26003 made
run-packaged-macos-testswrite aTestSummary.mdwhose first line is# ⚠️ <title>: Tests skipped, incorrect beta versionwhen it skips tests because the current OS is a beta build that doesn't match the expected build version.The CI summary parser (
tools/devops/automation/scripts/TestResults.psm1) only knew about the "all passed", "no tests selected" and "# Test results" formats, soGetPassedTeststhrewUnable to understand the test resultand the PR got:fire: Failed to compute test summariescomments instead of a proper summary.This PR:
GetPassedTests(treating it as zero passed tests, tracked via a new hidden$Skippedflag).:warning: ...: Tests skipped, incorrect beta version.message in the aggregated comment instead of the misleading "No tests selected.".🤖 Pull request created by Copilot