Skip to content

Blitzy: Add duration-aware test-file sharding to Vitest - #1

Closed
blitzy[bot] wants to merge 16 commits into
instance_647e6ade3b99523e3a0387a65fccfe918c331236from
blitzy-fda82f27-46ee-4ca4-b9f9-30f41987703c
Closed

blitzy[bot] wants to merge 16 commits into
instance_647e6ade3b99523e3a0387a65fccfe918c331236from
blitzy-fda82f27-46ee-4ca4-b9f9-30f41987703c

Conversation

@blitzy

@blitzy blitzy Bot commented Jul 16, 2026

Copy link
Copy Markdown

Adds a duration-aware test-file sharding subsystem to Vitest (packages/vitest), extending --shard with timing-driven strategies while keeping hash the default so existing users see byte-identical, zero-change behavior.

WHAT CHANGED

  • 12 new sequence.* config fields (shardStrategy, balanceShardsByTime, recordFileDurations, durationBasedSorting, durationHistoryTTL, durationHistoryPath, durationHistoryMaxRuns, durationSmoothing, shardAffinityRules, rebalanceThreshold, isolateSlowThreshold, durationFallbackStrategy) with defaults, throw-on-invalid validation, and balanceShardsByTime<->shardStrategy reconciliation.
  • 4 strategies via a BaseSequencer.shard() dispatcher: hash (unchanged), time (LPT bin-packing), round-robin (bouncing pointer), affinity (picomatch glob rules + LPT remainder, fall back to time).
  • Duration-history subsystem: 3 on-disk shapes (single/multi/legacy-numeric), TTL expiry, maxRuns retention, root-containment safety, null-on-corrupt; smoothing (latest/average/p95/median); safeguards isolateSlowThreshold, rebalanceThreshold warning, durationBasedSorting.
  • Non-fatal duration recording in runFiles() cleanup (integer ms, per-shard sidecar). Full worker propagation of all 12 fields; type-derived CLI surface satisfied (tsc gate).
  • New helpers: duration-history.ts, duration-smoothing.ts, shard-affinity.ts, shard-analytics.ts, lpt.ts. Docs (docs/config/sequence.md) + feat: changeset.

VALIDATION (independently re-verified)

  • tsc -p tsconfig.check.json --noEmit: 0 errors.
  • 882 in-scope/feature tests pass, 0 failures (sequencers 294/294, e2e recording 4/4, CLI 87, serialize/inject/provide 270, backward-compat/exports 227). Lint clean. No dependency changes.
  • Built CLI exercised end-to-end: all 4 strategies + recording feedback loop + validate-throw + rebalance + isolate-slow. Default hash byte-identical.

SCOPE: 19 files, +3987/-65. Confined to packages/vitest + docs + test.

REMAINING (22h, path-to-production, no feature rework): human review + revision (10h), full-suite CI on networked infra to clear 2 out-of-scope environmental failures (3h), staging shard-balance validation + threshold tuning (7h), release mechanics (2h).

NOTE: 2 failures in the full test/config suite are environmental (missing Playwright Chromium binary; no ambient TTY) in out-of-scope files where the feature is dormant — not defects.

blitzyai added 16 commits July 16, 2026 00:54
Introduce the configuration contract and persistence foundation for
distributing test files across `--shard` partitions by their historical
run time, alongside the existing, unchanged, default `hash` strategy.

- Declare twelve new `sequence.*` options on the user-facing
  `SequenceOptions` and on the resolved config: `shardStrategy`,
  `balanceShardsByTime`, `recordFileDurations`, `durationBasedSorting`,
  `durationHistoryTTL`, `durationHistoryPath`, `durationHistoryMaxRuns`,
  `durationSmoothing`, `shardAffinityRules`, `rebalanceThreshold`,
  `isolateSlowThreshold`, and `durationFallbackStrategy`, plus the
  supporting `ShardStrategy`, `DurationSmoothing`,
  `DurationFallbackStrategy`, and `ShardAffinityRule` types.
- Register the new options in the CLI option table so they are available
  as `--sequence.*` flags.
- Add a duration-history persistence helper that tolerantly reads and
  migrates single-observation, multi-observation, and legacy numeric
  entries with TTL filtering, and writes per-file durations with
  retention capping and parent-directory creation.
…g options

Wire up the configuration surface for duration-aware test-file sharding
in the `sequence` namespace so the sharding decision is consistent from
config resolution through to worker processes.

- Resolve defaults for the twelve sharding options (shardStrategy,
  balanceShardsByTime, recordFileDurations, durationBasedSorting,
  durationHistoryTTL, durationHistoryPath, durationHistoryMaxRuns,
  durationSmoothing, shardAffinityRules, rebalanceThreshold,
  isolateSlowThreshold, durationFallbackStrategy) during configuration
  resolution, validate each field and throw on invalid input rather than
  silently coercing, and cross-resolve balanceShardsByTime against
  shardStrategy (opt into 'time' when no strategy is explicitly set;
  otherwise force balanceShardsByTime off).
- Serialize all twelve resolved options into the worker configuration so
  every worker computes the same file-to-shard assignment.
- Align the CLI descriptions and argument placeholders for the twelve
  sequence options with their documented behavior.

The default shardStrategy remains 'hash', so projects that do not opt in
observe identical shard assignments.
…rker propagation, and history I/O

Address 8 code-review findings (7 Major, 1 Minor) in the duration-aware
test-file sharding subsystem.

resolveConfig.ts (R1-R4):
- R1: apply sequence defaults only on `undefined` (no null coercion) and
  validate every field against its exact runtime type, incl. strict
  typeof==='boolean' guards for balanceShardsByTime/recordFileDurations/
  durationBasedSorting; invalid values throw naming the exact sequence.* key.
- R2: validate rebalanceThreshold as a real finite number in [0,1]
  (no coercive relational comparison).
- R3 (CWE-22): reject absolute and root-escaping ('..'/'../'-prefixed)
  durationHistoryPath via pathe.isAbsolute + normalize.
- R4 (CWE-532): affinity-rule validation reports only the rule index and
  offending field/type; never stringifies the rule object (no secret leak).

serializeConfig.ts + runtime/config.ts (S1-S2):
- S2: extend SerializedConfig.sequence with the twelve new fields (inline
  literal unions to keep runtime/ free of any node/ import) as the type-level
  root cause, then type the serializer against the complete contract.
- S1: source the twelve new fields from the project-scoped config.sequence
  (keeping the original five from globalConfig.sequence) so workers receive
  per-project sharding inputs.

duration-history.ts (H1-H2):
- H1 (CWE-20): validate on-disk entries from `unknown`; require finite,
  nonnegative duration AND recordedAt; default recordedAt only when absent;
  build fresh observation objects; clamp non-finite/negative durations on
  write so JSON never emits null.
- H2 (CWE-1321): use Object.create(null) dictionaries for read/merge/write
  so a hostile `__proto__` key cannot pollute Object.prototype or crash the
  writer.

Validation: tsc -p tsconfig.check.json --noEmit clean; eslint clean;
test/config override+pool+cli-config 48/48; test/core sequencers 30/30;
built CLI loads. No behavioral change for the default shardStrategy: 'hash'.
The eight scalar sequence.* validators (shardStrategy, durationSmoothing,
durationFallbackStrategy, durationHistoryTTL, durationHistoryMaxRuns,
rebalanceThreshold, isolateSlowThreshold, durationHistoryPath) interpolated
the raw invalid value directly into their error-message template literals.
When a Symbol reached a validator (via untyped JS config or an 'as any' cast),
template-literal interpolation threw the opaque engine error
'TypeError: Cannot convert a Symbol value to a string', masking the intended
descriptive 'Invalid sequence.<key>' message.

Wrap each interpolated value with String(), mirroring the existing
shardAffinityRules[].shardIndex handling. String() renders a Symbol as
'Symbol(x)' and is byte-identical to template interpolation for all other
types, so error messages are unchanged for numbers, strings, null, objects,
etc. Every field now reports a descriptive, key-naming error.

Adds a parameterized regression test in test/config/test/public.test.ts
covering all eight scalar fields.

Resolves QA Issue #1 (checkpoint: packages/vitest/src/node/cli/cli-config.ts).
…per-file durations

Support duration-aware test-file sharding with two internal sequencing helpers
and a duration-recording hook:

- duration-smoothing: reduce a file's historical duration observations to a
  single value via the 'latest', 'average', 'p95', or 'median' strategy.
- shard-affinity: assign files to shards from glob-based affinity rules
  (picomatch, first match wins, shard index clamped to the shard count) and
  distribute the unmatched remainder using longest-processing-time bin-packing,
  signalling a fall-back when no rule matches any file.
- core: in the run cleanup phase, when sequence.recordFileDurations is enabled,
  persist per-file execution durations (slash-normalized root-relative keys,
  rounded to integer milliseconds) to the duration-history file. Recording is
  best-effort and never fails the run; default configuration is unchanged.
…options

Add a shard-load analytics helper that warns when the load imbalance
between shards (minLoad / maxLoad) falls below the configured
sequence.rebalanceThreshold, reporting the observed ratio and the
threshold so the distribution can be tuned for duration-aware sharding.

Document the duration-aware sharding options on the sequence configuration
page, covering the shard strategy, time balancing, duration history and
smoothing, affinity rules, the isolate-slow and rebalance thresholds, and
the fallback strategy, each with its type, default, and CLI form.
…encer

Turn BaseSequencer.shard() into a strategy dispatcher supporting hash
(default, byte-identical to the historical algorithm), time (LPT
bin-packing), round-robin (bouncing pointer), and affinity (picomatch
glob pinning with LPT remainder), plus durationFallbackStrategy
(hash|equal-split) when no history is available, isolateSlowThreshold
one-per-shard distribution, and rebalanceThreshold analytics warnings.
Add a durationBasedSorting early branch to sort() (duration DESC,
files absent from history last). The hash path performs no history
I/O and no analytics side effects, preserving backward compatibility
absolutely. calculateShardRange(), the constructor, and the existing
sort() comparator are unchanged.
Addresses the checkpoint-2 review (1 Critical scope violation, 8 Major, 3 Minor)
for the duration-aware test-file sharding feature.

SCOPE-1 (Critical): revert unauthorized Symbol test added to
  test/config/test/public.test.ts (out of AAP scope); the in-scope String()
  validation fix in resolveConfig.ts is retained.

F1  (Major): shard, sort and duration recording now use per-PROJECT config
  (spec.project.config.sequence + root) instead of the global/root config, so
  workspace projects key/write history relative to their own roots.
F2  (Major): fall back to durationFallbackStrategy when history is null OR has
  zero usable (non-expired) observations, not only when the file is missing.
F3  (Major): recording intersects state.getFiles() with the current run and
  writes only real, finite, non-negative measurements (real 0 kept, unmeasured
  omitted rather than coerced to 0).
F4  (Major): run cleanup inside an inner try and recording inside an inner
  finally, so recording always runs and any cleanup error still propagates.
F5  (Major): duration-history writer is now crash/concurrency safe — atomic
  temp-file + rename, a best-effort inter-process lock with stale-lock steal,
  and non-fatal recording in core.ts.
F6  (Major): read the history once per project and precompute each spec's key
  and smoothed duration (no per-comparator recomputation).
F7  (Major): force sequence.durationBasedSorting to false when RandomSequencer
  is selected (file shuffle takes precedence) during config resolution.
F8  (Major): validate affinity rules at resolution time (reject empty/overlong
  patterns, compile via picomatch with indexed errors) and harden
  shard-affinity so a malformed/hostile pattern degrades to never-match rather
  than throwing.
F9  (Minor): use Object.create(null) for per-project duration maps to prevent
  prototype-polluting keys (e.g. __proto__).
F10 (Minor): document exact smoothing formulas, round-robin boundary
  duplication, on-disk history shapes + legacy migration, empty/expired
  fallback, isolate-slow last-shard overflow, option precedence and the
  recording lifecycle in docs/config/sequence.md.
F11 (Minor): clamp durations to Number.MAX_SAFE_INTEGER and guard non-finite
  values across smoothing, analytics, the history writer and LPT load sums.

Also adds an explicit type annotation to the exported MAX_DURATION constant so
the --isolatedDeclarations build gate passes.
…oad scan

Add a thorough test suite for the duration-aware sharding engine in
BaseSequencer, exercising every distribution strategy — hash (byte-identical
to the historical behaviour), time (LPT bin-packing with lowest-index and
path-ascending tie-breaks), round-robin (bouncing pointer), and affinity
(picomatch pinning with a time fallback when no rule matches) — plus the
equal-split and hash duration fallbacks, duration smoothing, legacy/TTL
history handling, isolate-slow redistribution, the rebalance-threshold
warning tokens, and duration-based sorting.

Compute the per-shard load minimum and maximum in a single pass in the
rebalance analytics so a very high --shard count no longer risks a
"Maximum call stack size exceeded" RangeError from spreading a large loads
array onto Math.min/Math.max, while still coercing non-finite loads to keep
the imbalance ratio real so the warning continues to fire.
Extend the sequencer unit suite with comprehensive coverage for the new
duration-aware sharding feature while keeping all existing tests intact
(including the backward-compat hash-slice test.each proof):

- shard strategies: hash (default, equal-count preserved), time (LPT +
  lowest-index tie-break), round-robin (bouncing pointer), affinity
  (picomatch first-match + clamp + LPT remainder), and affinity->time
  fallback when no rule matches
- fallbacks with no history: equal-split and hash (via readDurationHistory
  returning null)
- isolateSlowThreshold (one-per-shard + overflow into last shard)
- rebalanceThreshold warning via logger (ratio/threshold tokens) + negatives
- durationBasedSorting in sort() (duration desc, absent files last)
- duration-history read/write/TTL directly (single/multi/legacy shapes,
  corrupt/missing -> null, TTL filtering, maxRuns capping, parent-dir creation)
- duration smoothing directly (latest/average/p95/median/empty)
- resolveConfig validation (throw-on-invalid) and balanceShardsByTime /
  shardStrategy cross-field reconciliation

Uses real temp files (no module mocking) with per-test cleanup; assertions
are deterministic.
Resolve all 14 code-review findings (1 Critical, 8 Major, 5 Minor) for the
duration-aware test-file sharding feature.

Sharding engine & history:
- F9: extract a shared deterministic LPT min-heap primitive (sequencers/lpt.ts)
  used by time LPT, isolate-slow remainder, and affinity remainder (removes the
  reachable O(n^2) triplicate shard scans; ties -> lowest-indexed shard).
- F8: aggregate rebalance analytics ONCE over final shard loads; the duration-
  aware path returns {specs, loads, rebalanceThreshold} instead of warning per
  project (fixes multi-project false-positive warnings).
- F1: gate duration-based sorting on each project's resolved flag (not the
  global flag) and include TTL in the history cache key.
- F12: hash-first fast path in BaseSequencer.shard() and skip the recording
  helper entirely when no project opts in (byte-identical default hash path).
- F3: remove non-specified clamping/saturation from history, smoothing, and load
  accumulation so exact accepted finite values/formulas are preserved.
- F7: compact {duration, recordedAt} history entries now require BOTH finite
  non-negative fields; only the legacy bare-number form is migrated to the
  immortal recordedAt:0 sentinel.

Security & filesystem safety:
- F13 (Critical): compile affinity globs with picomatch {noextglob:true} and
  reject extglob-opener / POSIX '[[:' patterns at config resolution and match
  time (defense-in-depth against CVE-2026-33671/33672); dependency version bump
  deferred per AAP 0.3/0.6.2.
- F14: enforce real-path containment beneath the project root before any write
  and create the temp file exclusively ('wx') so symlinked paths cannot be
  followed/overwritten.
- F2: ownership-token advisory lock (pid+ts+rand); release only on token match,
  steal only when stale, never proceed unlocked, skip write on budget exhaust.
- F11: wrap temp write+rename in an outer cleanup finally that unlinks the temp
  file on every failure path.

Lifecycle & docs:
- F10: per-project try/catch around each history write so one project's failure
  cannot suppress later projects' recording.
- F6: correct docs to state per-project history is independent only with
  distinct roots OR distinct paths (same root + default path share the file).

Tests (F4/F5): add failure-sensitive, direct coverage for serializer transport,
history edge cases/concurrency/containment, per-project sorting, multi-project
rebalance, affinity security, write-path, seed sensitivity, timestamp ties, CLI
surface, and exhaustive runtime validation; strengthen deterministic fixtures
to assert exact frozen partitions. Suite: 138 -> 240 passing.
Add an integration test that runs a real Vitest child process to verify
the sequence.recordFileDurations recording lifecycle wired into the run
cleanup phase:

- durationHistoryMaxRuns === 1 persists per-file durations in the compact
  { duration, recordedAt } shape with integer millisecond durations and a
  real wall-clock timestamp, keyed by slash-normalized project-relative paths.
- durationHistoryMaxRuns > 1 writes the { observations } shape and preserves
  (migrating the legacy numeric form) entries for files outside the current run.
- A failure to write the history file is best-effort and never fails the run:
  the suite still exits 0 and the pre-existing target is left intact.

This exercises the full pipeline (config resolution through history
persistence) that the existing unit-level write-path tests cover only in
isolation.
…to 4.0.5; document path containment

QA remediation for duration-aware sharding:
- SEC-1: refresh pnpm-lock.yaml picomatch 4.0.3 -> 4.0.5 (within declared ^4.0.3, package.json unchanged), resolving CVE-2026-33671 (ReDoS extglob) and CVE-2026-33672 (POSIX injection) on the shard-affinity path
- TEST-5: add strict-boundary regression tests — isolateSlowThreshold equality (strict >), rebalance ratio-equals-threshold (strict <), and hash strategy bypassing rebalance analytics
- INFO-4: document that durationHistoryPath rejects absolute paths and parent-directory ('..') traversal
Addresses 5 QA findings (2 Major, 3 Minor) in the duration-aware test-file sharding feature. Default 'hash' strategy remains byte-identical; all fixes verified statically (typecheck/lint/build exit 0) and at runtime (tsx/CLI).

QA-P6-STAGGERED-1 (Major, functional): sequential --shard jobs with recordFileDurations + a duration-aware strategy mutated the shared base history file mid-run, so later shards read a changed partition basis and skipped/duplicated files. core.ts now routes a sharded run's write to a per-shard sidecar (duration-history.shard-<index>-of-<count>.json) via the new shardHistoryPath helper, freezing the base file (partition basis) for the whole logical run. Non-sharded runs are unchanged.

QA-P6-DOCS-CLI-1 (Major, docs/CI): regenerated docs/guide/cli-generated.md so the 11 CLI-exposed sequence.* options are documented (shardAffinityRules stays config-only). Regeneration is idempotent (git diff clean on re-run).

QA-P6-SYMLINK-1 (Minor, security/fs): readDurationHistory containment only checked the parent dir, so a leaf symlink escaping the project root was followed on read. Added isReadPathWithinRoot which follows the leaf via realpathSync and requires the real target within the real root; the atomic temp+rename write path is intentionally unchanged.

QA-P4-CLI-1 (Minor, config boundary): CAC's dotted-key parser could assemble a config-only sequence.shardAffinityRules array from --sequence.shardAffinityRules.0.pattern=... Added assertNoCliOnlySequenceOptions, enforced in both parseCLI and normalizeCliOptions, throwing a clear error that points to the config file. Other CLI sequence flags and config-file usage are unaffected.

QA-P10-OVERFLOW-1 (Minor, determinism): pathological but individually finite observations (e.g. 1e308) overflowed average/median smoothing to Infinity, and subtractive comparators evaluated Infinity-Infinity to NaN, defeating the mandated path-ascending tie-break. duration-smoothing.ts now uses finite-guarded overflow-safe average/median (byte-identical for normal values); BaseSequencer.ts and shard-affinity.ts use relational (not subtractive) duration comparators.

Tests: extend test/core/test/sequencers.test.ts (overflow determinism, finite smoothing, leaf-symlink refusal, shardHistoryPath naming), test/core/test/cli-test.test.ts (config-only CLI rejection), and test/config/test/shard-record-durations.test.ts (sharded sidecar leaves base frozen).
…shard sidecar

Addresses QA FINAL report findings (HEAD 516555c):

Finding #1 (MINOR, blocking): the duration-aware sharding feature had bumped
picomatch 4.0.3 -> 4.0.5 in pnpm-lock.yaml, contradicting the AAP's binding
'no dependency changes' rule (0.3/0.6.2/0.7) and the in-source comments that
state picomatch is pinned at 4.0.3. Revert pnpm-lock.yaml to the base
resolution so no dependency is added/updated. The picomatch CVEs remain
mitigated in code, version-independently, via isUnsafeAffinityPattern() +
AFFINITY_PICOMATCH_OPTIONS (noextglob) in shard-affinity.ts, so reverting
does not reintroduce an exploitable path. The three source comments claiming
'pinned 4.0.3' are now factually accurate again.

Finding #2 (INFO): document the per-shard duration-history sidecar behavior
in docs/config/sequence.md (recordFileDurations 'deliberate limits'): when a
run is sharded, durations are written to duration-history.shard-<i>-of-<n>.json
beside the base file so each shard partitions from an identical frozen
snapshot; non-sharded runs write the base file directly.

Verification: tsc --noEmit, eslint, vitest build, and vitepress docs build all
pass; sequencers (294) + cli + shard/record e2e (12) suites green;
git diff <base> -- pnpm-lock.yaml is empty (zero dependency drift).
@blitzy blitzy Bot closed this Jul 20, 2026
blitzy Bot pushed a commit that referenced this pull request Jul 21, 2026
…ts final order

QA finding (dest Issue #1, CRITICAL): durationBasedSorting was applied only inside BaseSequencer.shard(), but pool.ts always calls sequencer.sort() after shard() and consumes sort()'s output as the definitive execution order. sort() had no duration awareness, so its file-size/cache heuristic silently overwrote the duration ordering and durationBasedSorting had no observable effect at runtime.

Fix: add a shared loadSmoothedDurations() helper (single source of truth for per-file smoothed durations) plus historyKeyFor(); gate sort() on sequence.durationBasedSorting (default false => byte-for-byte unchanged) and apply a descending-duration comparison after the isolate checks and before the cache/size heuristics. shard()'s existing duration sort is retained and now reuses the shared helper so keys never diverge.

Verification: tsc typecheck 0 errors; eslint 0 violations; sequencers 30/30, duration-sharding 96/96 (incl. 3 new add-only sort()-through cases), sequence-shuffle 14/14, pool 15/15. Runtime re-verified via real Vitest CLI: final module order == descending smoothed duration, stable across repeated runs; hash default partition, time/LPT membership, and rebalance warning tokens all unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant