Repository navigation
Blitzy: Add duration-aware test-file sharding to Vitest - #1
Closed
blitzy[bot] wants to merge 16 commits into
Closed
blitzy[bot] wants to merge 16 commits into
blitzy[bot] wants to merge 16 commits into
Conversation
Introduce the configuration contract and persistence foundation for distributing test files across `--shard` partitions by their historical run time, alongside the existing, unchanged, default `hash` strategy. - Declare twelve new `sequence.*` options on the user-facing `SequenceOptions` and on the resolved config: `shardStrategy`, `balanceShardsByTime`, `recordFileDurations`, `durationBasedSorting`, `durationHistoryTTL`, `durationHistoryPath`, `durationHistoryMaxRuns`, `durationSmoothing`, `shardAffinityRules`, `rebalanceThreshold`, `isolateSlowThreshold`, and `durationFallbackStrategy`, plus the supporting `ShardStrategy`, `DurationSmoothing`, `DurationFallbackStrategy`, and `ShardAffinityRule` types. - Register the new options in the CLI option table so they are available as `--sequence.*` flags. - Add a duration-history persistence helper that tolerantly reads and migrates single-observation, multi-observation, and legacy numeric entries with TTL filtering, and writes per-file durations with retention capping and parent-directory creation.
…g options Wire up the configuration surface for duration-aware test-file sharding in the `sequence` namespace so the sharding decision is consistent from config resolution through to worker processes. - Resolve defaults for the twelve sharding options (shardStrategy, balanceShardsByTime, recordFileDurations, durationBasedSorting, durationHistoryTTL, durationHistoryPath, durationHistoryMaxRuns, durationSmoothing, shardAffinityRules, rebalanceThreshold, isolateSlowThreshold, durationFallbackStrategy) during configuration resolution, validate each field and throw on invalid input rather than silently coercing, and cross-resolve balanceShardsByTime against shardStrategy (opt into 'time' when no strategy is explicitly set; otherwise force balanceShardsByTime off). - Serialize all twelve resolved options into the worker configuration so every worker computes the same file-to-shard assignment. - Align the CLI descriptions and argument placeholders for the twelve sequence options with their documented behavior. The default shardStrategy remains 'hash', so projects that do not opt in observe identical shard assignments.
…rker propagation, and history I/O
Address 8 code-review findings (7 Major, 1 Minor) in the duration-aware
test-file sharding subsystem.
resolveConfig.ts (R1-R4):
- R1: apply sequence defaults only on `undefined` (no null coercion) and
validate every field against its exact runtime type, incl. strict
typeof==='boolean' guards for balanceShardsByTime/recordFileDurations/
durationBasedSorting; invalid values throw naming the exact sequence.* key.
- R2: validate rebalanceThreshold as a real finite number in [0,1]
(no coercive relational comparison).
- R3 (CWE-22): reject absolute and root-escaping ('..'/'../'-prefixed)
durationHistoryPath via pathe.isAbsolute + normalize.
- R4 (CWE-532): affinity-rule validation reports only the rule index and
offending field/type; never stringifies the rule object (no secret leak).
serializeConfig.ts + runtime/config.ts (S1-S2):
- S2: extend SerializedConfig.sequence with the twelve new fields (inline
literal unions to keep runtime/ free of any node/ import) as the type-level
root cause, then type the serializer against the complete contract.
- S1: source the twelve new fields from the project-scoped config.sequence
(keeping the original five from globalConfig.sequence) so workers receive
per-project sharding inputs.
duration-history.ts (H1-H2):
- H1 (CWE-20): validate on-disk entries from `unknown`; require finite,
nonnegative duration AND recordedAt; default recordedAt only when absent;
build fresh observation objects; clamp non-finite/negative durations on
write so JSON never emits null.
- H2 (CWE-1321): use Object.create(null) dictionaries for read/merge/write
so a hostile `__proto__` key cannot pollute Object.prototype or crash the
writer.
Validation: tsc -p tsconfig.check.json --noEmit clean; eslint clean;
test/config override+pool+cli-config 48/48; test/core sequencers 30/30;
built CLI loads. No behavioral change for the default shardStrategy: 'hash'.
The eight scalar sequence.* validators (shardStrategy, durationSmoothing, durationFallbackStrategy, durationHistoryTTL, durationHistoryMaxRuns, rebalanceThreshold, isolateSlowThreshold, durationHistoryPath) interpolated the raw invalid value directly into their error-message template literals. When a Symbol reached a validator (via untyped JS config or an 'as any' cast), template-literal interpolation threw the opaque engine error 'TypeError: Cannot convert a Symbol value to a string', masking the intended descriptive 'Invalid sequence.<key>' message. Wrap each interpolated value with String(), mirroring the existing shardAffinityRules[].shardIndex handling. String() renders a Symbol as 'Symbol(x)' and is byte-identical to template interpolation for all other types, so error messages are unchanged for numbers, strings, null, objects, etc. Every field now reports a descriptive, key-naming error. Adds a parameterized regression test in test/config/test/public.test.ts covering all eight scalar fields. Resolves QA Issue #1 (checkpoint: packages/vitest/src/node/cli/cli-config.ts).
…per-file durations Support duration-aware test-file sharding with two internal sequencing helpers and a duration-recording hook: - duration-smoothing: reduce a file's historical duration observations to a single value via the 'latest', 'average', 'p95', or 'median' strategy. - shard-affinity: assign files to shards from glob-based affinity rules (picomatch, first match wins, shard index clamped to the shard count) and distribute the unmatched remainder using longest-processing-time bin-packing, signalling a fall-back when no rule matches any file. - core: in the run cleanup phase, when sequence.recordFileDurations is enabled, persist per-file execution durations (slash-normalized root-relative keys, rounded to integer milliseconds) to the duration-history file. Recording is best-effort and never fails the run; default configuration is unchanged.
…options Add a shard-load analytics helper that warns when the load imbalance between shards (minLoad / maxLoad) falls below the configured sequence.rebalanceThreshold, reporting the observed ratio and the threshold so the distribution can be tuned for duration-aware sharding. Document the duration-aware sharding options on the sequence configuration page, covering the shard strategy, time balancing, duration history and smoothing, affinity rules, the isolate-slow and rebalance thresholds, and the fallback strategy, each with its type, default, and CLI form.
…encer Turn BaseSequencer.shard() into a strategy dispatcher supporting hash (default, byte-identical to the historical algorithm), time (LPT bin-packing), round-robin (bouncing pointer), and affinity (picomatch glob pinning with LPT remainder), plus durationFallbackStrategy (hash|equal-split) when no history is available, isolateSlowThreshold one-per-shard distribution, and rebalanceThreshold analytics warnings. Add a durationBasedSorting early branch to sort() (duration DESC, files absent from history last). The hash path performs no history I/O and no analytics side effects, preserving backward compatibility absolutely. calculateShardRange(), the constructor, and the existing sort() comparator are unchanged.
Addresses the checkpoint-2 review (1 Critical scope violation, 8 Major, 3 Minor) for the duration-aware test-file sharding feature. SCOPE-1 (Critical): revert unauthorized Symbol test added to test/config/test/public.test.ts (out of AAP scope); the in-scope String() validation fix in resolveConfig.ts is retained. F1 (Major): shard, sort and duration recording now use per-PROJECT config (spec.project.config.sequence + root) instead of the global/root config, so workspace projects key/write history relative to their own roots. F2 (Major): fall back to durationFallbackStrategy when history is null OR has zero usable (non-expired) observations, not only when the file is missing. F3 (Major): recording intersects state.getFiles() with the current run and writes only real, finite, non-negative measurements (real 0 kept, unmeasured omitted rather than coerced to 0). F4 (Major): run cleanup inside an inner try and recording inside an inner finally, so recording always runs and any cleanup error still propagates. F5 (Major): duration-history writer is now crash/concurrency safe — atomic temp-file + rename, a best-effort inter-process lock with stale-lock steal, and non-fatal recording in core.ts. F6 (Major): read the history once per project and precompute each spec's key and smoothed duration (no per-comparator recomputation). F7 (Major): force sequence.durationBasedSorting to false when RandomSequencer is selected (file shuffle takes precedence) during config resolution. F8 (Major): validate affinity rules at resolution time (reject empty/overlong patterns, compile via picomatch with indexed errors) and harden shard-affinity so a malformed/hostile pattern degrades to never-match rather than throwing. F9 (Minor): use Object.create(null) for per-project duration maps to prevent prototype-polluting keys (e.g. __proto__). F10 (Minor): document exact smoothing formulas, round-robin boundary duplication, on-disk history shapes + legacy migration, empty/expired fallback, isolate-slow last-shard overflow, option precedence and the recording lifecycle in docs/config/sequence.md. F11 (Minor): clamp durations to Number.MAX_SAFE_INTEGER and guard non-finite values across smoothing, analytics, the history writer and LPT load sums. Also adds an explicit type annotation to the exported MAX_DURATION constant so the --isolatedDeclarations build gate passes.
…oad scan Add a thorough test suite for the duration-aware sharding engine in BaseSequencer, exercising every distribution strategy — hash (byte-identical to the historical behaviour), time (LPT bin-packing with lowest-index and path-ascending tie-breaks), round-robin (bouncing pointer), and affinity (picomatch pinning with a time fallback when no rule matches) — plus the equal-split and hash duration fallbacks, duration smoothing, legacy/TTL history handling, isolate-slow redistribution, the rebalance-threshold warning tokens, and duration-based sorting. Compute the per-shard load minimum and maximum in a single pass in the rebalance analytics so a very high --shard count no longer risks a "Maximum call stack size exceeded" RangeError from spreading a large loads array onto Math.min/Math.max, while still coercing non-finite loads to keep the imbalance ratio real so the warning continues to fire.
Extend the sequencer unit suite with comprehensive coverage for the new duration-aware sharding feature while keeping all existing tests intact (including the backward-compat hash-slice test.each proof): - shard strategies: hash (default, equal-count preserved), time (LPT + lowest-index tie-break), round-robin (bouncing pointer), affinity (picomatch first-match + clamp + LPT remainder), and affinity->time fallback when no rule matches - fallbacks with no history: equal-split and hash (via readDurationHistory returning null) - isolateSlowThreshold (one-per-shard + overflow into last shard) - rebalanceThreshold warning via logger (ratio/threshold tokens) + negatives - durationBasedSorting in sort() (duration desc, absent files last) - duration-history read/write/TTL directly (single/multi/legacy shapes, corrupt/missing -> null, TTL filtering, maxRuns capping, parent-dir creation) - duration smoothing directly (latest/average/p95/median/empty) - resolveConfig validation (throw-on-invalid) and balanceShardsByTime / shardStrategy cross-field reconciliation Uses real temp files (no module mocking) with per-test cleanup; assertions are deterministic.
Resolve all 14 code-review findings (1 Critical, 8 Major, 5 Minor) for the
duration-aware test-file sharding feature.
Sharding engine & history:
- F9: extract a shared deterministic LPT min-heap primitive (sequencers/lpt.ts)
used by time LPT, isolate-slow remainder, and affinity remainder (removes the
reachable O(n^2) triplicate shard scans; ties -> lowest-indexed shard).
- F8: aggregate rebalance analytics ONCE over final shard loads; the duration-
aware path returns {specs, loads, rebalanceThreshold} instead of warning per
project (fixes multi-project false-positive warnings).
- F1: gate duration-based sorting on each project's resolved flag (not the
global flag) and include TTL in the history cache key.
- F12: hash-first fast path in BaseSequencer.shard() and skip the recording
helper entirely when no project opts in (byte-identical default hash path).
- F3: remove non-specified clamping/saturation from history, smoothing, and load
accumulation so exact accepted finite values/formulas are preserved.
- F7: compact {duration, recordedAt} history entries now require BOTH finite
non-negative fields; only the legacy bare-number form is migrated to the
immortal recordedAt:0 sentinel.
Security & filesystem safety:
- F13 (Critical): compile affinity globs with picomatch {noextglob:true} and
reject extglob-opener / POSIX '[[:' patterns at config resolution and match
time (defense-in-depth against CVE-2026-33671/33672); dependency version bump
deferred per AAP 0.3/0.6.2.
- F14: enforce real-path containment beneath the project root before any write
and create the temp file exclusively ('wx') so symlinked paths cannot be
followed/overwritten.
- F2: ownership-token advisory lock (pid+ts+rand); release only on token match,
steal only when stale, never proceed unlocked, skip write on budget exhaust.
- F11: wrap temp write+rename in an outer cleanup finally that unlinks the temp
file on every failure path.
Lifecycle & docs:
- F10: per-project try/catch around each history write so one project's failure
cannot suppress later projects' recording.
- F6: correct docs to state per-project history is independent only with
distinct roots OR distinct paths (same root + default path share the file).
Tests (F4/F5): add failure-sensitive, direct coverage for serializer transport,
history edge cases/concurrency/containment, per-project sorting, multi-project
rebalance, affinity security, write-path, seed sensitivity, timestamp ties, CLI
surface, and exhaustive runtime validation; strengthen deterministic fixtures
to assert exact frozen partitions. Suite: 138 -> 240 passing.
Add an integration test that runs a real Vitest child process to verify
the sequence.recordFileDurations recording lifecycle wired into the run
cleanup phase:
- durationHistoryMaxRuns === 1 persists per-file durations in the compact
{ duration, recordedAt } shape with integer millisecond durations and a
real wall-clock timestamp, keyed by slash-normalized project-relative paths.
- durationHistoryMaxRuns > 1 writes the { observations } shape and preserves
(migrating the legacy numeric form) entries for files outside the current run.
- A failure to write the history file is best-effort and never fails the run:
the suite still exits 0 and the pre-existing target is left intact.
This exercises the full pipeline (config resolution through history
persistence) that the existing unit-level write-path tests cover only in
isolation.
…to 4.0.5; document path containment QA remediation for duration-aware sharding: - SEC-1: refresh pnpm-lock.yaml picomatch 4.0.3 -> 4.0.5 (within declared ^4.0.3, package.json unchanged), resolving CVE-2026-33671 (ReDoS extglob) and CVE-2026-33672 (POSIX injection) on the shard-affinity path - TEST-5: add strict-boundary regression tests — isolateSlowThreshold equality (strict >), rebalance ratio-equals-threshold (strict <), and hash strategy bypassing rebalance analytics - INFO-4: document that durationHistoryPath rejects absolute paths and parent-directory ('..') traversal
Addresses 5 QA findings (2 Major, 3 Minor) in the duration-aware test-file sharding feature. Default 'hash' strategy remains byte-identical; all fixes verified statically (typecheck/lint/build exit 0) and at runtime (tsx/CLI). QA-P6-STAGGERED-1 (Major, functional): sequential --shard jobs with recordFileDurations + a duration-aware strategy mutated the shared base history file mid-run, so later shards read a changed partition basis and skipped/duplicated files. core.ts now routes a sharded run's write to a per-shard sidecar (duration-history.shard-<index>-of-<count>.json) via the new shardHistoryPath helper, freezing the base file (partition basis) for the whole logical run. Non-sharded runs are unchanged. QA-P6-DOCS-CLI-1 (Major, docs/CI): regenerated docs/guide/cli-generated.md so the 11 CLI-exposed sequence.* options are documented (shardAffinityRules stays config-only). Regeneration is idempotent (git diff clean on re-run). QA-P6-SYMLINK-1 (Minor, security/fs): readDurationHistory containment only checked the parent dir, so a leaf symlink escaping the project root was followed on read. Added isReadPathWithinRoot which follows the leaf via realpathSync and requires the real target within the real root; the atomic temp+rename write path is intentionally unchanged. QA-P4-CLI-1 (Minor, config boundary): CAC's dotted-key parser could assemble a config-only sequence.shardAffinityRules array from --sequence.shardAffinityRules.0.pattern=... Added assertNoCliOnlySequenceOptions, enforced in both parseCLI and normalizeCliOptions, throwing a clear error that points to the config file. Other CLI sequence flags and config-file usage are unaffected. QA-P10-OVERFLOW-1 (Minor, determinism): pathological but individually finite observations (e.g. 1e308) overflowed average/median smoothing to Infinity, and subtractive comparators evaluated Infinity-Infinity to NaN, defeating the mandated path-ascending tie-break. duration-smoothing.ts now uses finite-guarded overflow-safe average/median (byte-identical for normal values); BaseSequencer.ts and shard-affinity.ts use relational (not subtractive) duration comparators. Tests: extend test/core/test/sequencers.test.ts (overflow determinism, finite smoothing, leaf-symlink refusal, shardHistoryPath naming), test/core/test/cli-test.test.ts (config-only CLI rejection), and test/config/test/shard-record-durations.test.ts (sharded sidecar leaves base frozen).
…shard sidecar Addresses QA FINAL report findings (HEAD 516555c): Finding #1 (MINOR, blocking): the duration-aware sharding feature had bumped picomatch 4.0.3 -> 4.0.5 in pnpm-lock.yaml, contradicting the AAP's binding 'no dependency changes' rule (0.3/0.6.2/0.7) and the in-source comments that state picomatch is pinned at 4.0.3. Revert pnpm-lock.yaml to the base resolution so no dependency is added/updated. The picomatch CVEs remain mitigated in code, version-independently, via isUnsafeAffinityPattern() + AFFINITY_PICOMATCH_OPTIONS (noextglob) in shard-affinity.ts, so reverting does not reintroduce an exploitable path. The three source comments claiming 'pinned 4.0.3' are now factually accurate again. Finding #2 (INFO): document the per-shard duration-history sidecar behavior in docs/config/sequence.md (recordFileDurations 'deliberate limits'): when a run is sharded, durations are written to duration-history.shard-<i>-of-<n>.json beside the base file so each shard partitions from an identical frozen snapshot; non-sharded runs write the base file directly. Verification: tsc --noEmit, eslint, vitest build, and vitepress docs build all pass; sequencers (294) + cli + shard/record e2e (12) suites green; git diff <base> -- pnpm-lock.yaml is empty (zero dependency drift).
blitzy Bot
pushed a commit
that referenced
this pull request
Jul 21, 2026
…ts final order QA finding (dest Issue #1, CRITICAL): durationBasedSorting was applied only inside BaseSequencer.shard(), but pool.ts always calls sequencer.sort() after shard() and consumes sort()'s output as the definitive execution order. sort() had no duration awareness, so its file-size/cache heuristic silently overwrote the duration ordering and durationBasedSorting had no observable effect at runtime. Fix: add a shared loadSmoothedDurations() helper (single source of truth for per-file smoothed durations) plus historyKeyFor(); gate sort() on sequence.durationBasedSorting (default false => byte-for-byte unchanged) and apply a descending-duration comparison after the isolate checks and before the cache/size heuristics. shard()'s existing duration sort is retained and now reuses the shared helper so keys never diverge. Verification: tsc typecheck 0 errors; eslint 0 violations; sequencers 30/30, duration-sharding 96/96 (incl. 3 new add-only sort()-through cases), sequence-shuffle 14/14, pool 15/15. Runtime re-verified via real Vitest CLI: final module order == descending smoothed duration, stable across repeated runs; hash default partition, time/LPT membership, and rebalance warning tokens all unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a duration-aware test-file sharding subsystem to Vitest (packages/vitest), extending --shard with timing-driven strategies while keeping hash the default so existing users see byte-identical, zero-change behavior.
WHAT CHANGED
VALIDATION (independently re-verified)
SCOPE: 19 files, +3987/-65. Confined to packages/vitest + docs + test.
REMAINING (22h, path-to-production, no feature rework): human review + revision (10h), full-suite CI on networked infra to clear 2 out-of-scope environmental failures (3h), staging shard-balance validation + threshold tuning (7h), release mechanics (2h).
NOTE: 2 failures in the full test/config suite are environmental (missing Playwright Chromium binary; no ambient TTY) in out-of-scope files where the feature is dormant — not defects.