Skip to content

test(cli): shift the releases.json fixture tags below the v0.1.0 line - #88

Merged
oskarhane merged 1 commit into
mainfrom
fix-upgrade-fixture-versions
Nov 9, 2026
Merged

oskarhane merged 1 commit into
mainfrom
fix-upgrade-fixture-versions

Conversation

@oskarhane

Copy link
Copy Markdown
Contributor

Unblocks every open PR — the storage shard's e2e_upgrade tests fail on main since the v0.1.0 release.

Root cause: the upgrade fixtures' latest tags (v0.10.0/v0.10.1) were chosen to 'predate every released build' — true at 0.16.x, inverted by the v0.1.0 renumbering. Semver-correctly, upgrade --check now reports 'upgrade available' where the tests assert the up-to-date path.

Fix: shift every fixture tag below v0.1.0, order preserved (CLI fixture latest → v0.0.9; publisher fixture pair window → v0.0.2–v0.0.11, prerelease slot → v0.0.12-rc.1). The bats publisher pair moves together so the exact-output compare holds; the skeleton compare reads structure only; parse-only unit fixtures keep their tags.

Verified locally: releases_json 9/9, e2e_upgrade 21/21, publish-releases-json.bats 18/18, skeleton compare, clippy, fmt.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Security review (security-review skill)

Reviewed the releases.json fixture tag shift across the .github/scripts/tests/ fixtures and bats assertions, the relate-cli test-only assertion updates, and the .sonic/ci.yml shard additions. Every change is test data, a test expectation, or CI test-partition membership: no production code, network surface, auth logic, secret handling, or dependency/lockfile changes. The upgrade signature-verification and tokenless-metadata assertions are untouched (they use separate v99.0.0 route fixtures), the publisher script is unchanged, and CI workflow permissions/secrets are not modified.

No issues found.

Security review: ✅ no issues found

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Rust review (rust-reviewer skill, full review)

The Rust side of this diff is test-data only: the releases.json fixture tags move below the v0.1.0 line, with the matching assertion strings updated in releases_json.rs and e2e_upgrade.rs. I reviewed ownership/borrowing, lifetimes, error handling (Result/Option), unsafe, and idiomatic patterns. No production code changed, and the updated assertions match the fixtures; the remaining v0.10.0 references are the deliberately parse-only MINIMAL_LATEST anchor, now documented as such.

No issues found.

Rust review: ✅ no issues found

@oskarhane
oskarhane force-pushed the fix-upgrade-fixture-versions branch from 404d455 to e785868 Compare November 9, 2026 14:42
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Stellar review (stellar-review skill)

Round 4 · head 388c5a86 · scope: 1/67 units (66 sealed from earlier rounds).

Still open from earlier rounds

  • [Medium] boundaries (std-5) F-1-1: surfaces/relate-cli/src/upgrade_cmd/releases_json.rs:411-414 — The two unknown keys are spliced into the fixture with replacen (lines 400-410), but this aggregate guard only proves the document changed at all. If either anchor stops matching while the other sti

Stellar review: ✅ no issues found

@oskarhane
oskarhane force-pushed the fix-upgrade-fixture-versions branch from e785868 to 7f8df31 Compare November 9, 2026 14:44
The v0.1.0 renumbering broke the e2e upgrade fixtures' premise: their
latest tags (v0.10.0/v0.10.1) were chosen to predate every released
build, which held at 0.16.x and inverts at 0.1.0 — semver-correctly,
--check then reports 'upgrade available' where the up-to-date path was
asserted, and every PR's storage shard goes red.

Shift every fixture tag below v0.1.0, order preserved (the CLI fixture's
latest is v0.0.9; the publisher fixture pair's window is v0.0.2–v0.0.11
with the reserved prerelease slot at v0.0.12-rc.1). The publisher bats
pair moves together so the exact-output compare holds; the skeleton
compare reads structure only. Parse-only unit fixtures keep their tags
(no version comparison there).

Verified: cargo test -p relate-cli releases_json (9/9), e2e_upgrade
(21/21), publish-releases-json.bats (18/18), compare-releases-json-
skeletons.py, clippy, fmt.
@oskarhane
oskarhane force-pushed the fix-upgrade-fixture-versions branch from 7f8df31 to 388c5a8 Compare November 9, 2026 14:50
@oskarhane
oskarhane marked this pull request as ready for review November 9, 2026 14:53
@oplane-bot

oplane-bot Bot commented Nov 9, 2026

Copy link
Copy Markdown

Oplane Security Review

Oplane reviewed this PR and did not identify any security requirements that need attention. Review the generated threat model.


Change comment settings · Set up Oplane in your agent · Workflow guidance

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

Loadtest baselines comparison

Note: loadtest comparison completed with no failing rows.


Loadtest baseline comparison

Frozen baselines vs current reports; bands match the compare-loadtests gate.

Status: ok

Pairs: fraud-loadtest: compared · identity-loadtest: compared

fraud-loadtest

baselines/fraud-loadtest.json → target/fraud-loadtest.json

Metric Baseline PR Δ Status
environment relate-loadtest-baseline/v1 relate-loadtest-baseline/v1 — match
errors 0 0 +0 ok
ops/s 3938.56 6922.81 +2984.25 (+75.8%) ok
p99 (worst: search_customer) 7.84 ms 7.36 ms -0.48 ms (-6.2%) ok
logical bytes 17,344,481 17,294,286 -50,195 (-0.3%) ok
Per-use-case p99 latency
Use case ops Baseline p99 PR p99 Δ Status
p2p_transfer 3,876 8.04 ms 6.98 ms -1.06 ms (-13.2%) ok
payroll_scatter 1,015 17.85 ms 10.18 ms -7.67 ms (-43.0%) ok
onboard_customer 887 8.53 ms 7.44 ms -1.09 ms (-12.8%) ok
mule_scatter 797 14.80 ms 9.05 ms -5.75 ms (-38.8%) ok
layering_cycle 846 15.10 ms 9.47 ms -5.63 ms (-37.3%) ok
recent_transfers 2,260 7.65 ms 6.35 ms -1.31 ms (-17.1%) ok
customer_portfolio 1,344 7.37 ms 6.15 ms -1.23 ms (-16.7%) ok
outgoing_volume 1,114 7.48 ms 6.55 ms -0.92 ms (-12.4%) ok
detect_fanout 861 25.78 ms 11.73 ms -14.05 ms (-54.5%) ok
detect_cycle 847 7.55 ms 5.91 ms -1.64 ms (-21.7%) ok
activity_page 1,131 7.48 ms 6.01 ms -1.48 ms (-19.7%) ok
search_customer 828 7.84 ms 7.36 ms -0.48 ms (-6.2%) ok
counterparties 1,101 7.45 ms 5.92 ms -1.53 ms (-20.5%) ok
payment_path 874 7.47 ms 6.35 ms -1.12 ms (-15.1%) ok
amount_band 867 7.30 ms 5.99 ms -1.31 ms (-18.0%) ok
flag_account 735 8.21 ms 6.37 ms -1.83 ms (-22.3%) ok
reverse_last_out 617 8.04 ms 6.44 ms -1.60 ms (-19.9%) ok

identity-loadtest

baselines/identity-loadtest.json → target/identity-loadtest.json

Metric Baseline PR Δ Status
environment relate-loadtest-baseline/v1 relate-loadtest-baseline/v1 — match
errors 0 0 +0 ok
ops/s 573.51 1026.59 +453.08 (+79.0%) ok
p99 (worst: group_roster) 7.72 ms 7.49 ms -0.23 ms (-3.0%) ok
logical bytes 7,167,614 7,165,975 -1,639 (-0.0%) ok
Per-use-case p99 latency
Use case ops Baseline p99 PR p99 Δ Status
provision_user 1,373 11.00 ms 8.64 ms -2.36 ms (-21.5%) ok
add_to_group 1,775 18.57 ms 11.93 ms -6.64 ms (-35.7%) ok
grant_role 1,079 18.52 ms 11.86 ms -6.66 ms (-35.9%) ok
attach_device 907 18.88 ms 12.18 ms -6.70 ms (-35.5%) ok
nested_grant 736 19.41 ms 11.95 ms -7.46 ms (-38.4%) ok
effective_access 2,173 25.25 ms 16.21 ms -9.03 ms (-35.8%) ok
org_walk 1,387 19.94 ms 11.25 ms -8.69 ms (-43.6%) ok
optional_manager 1,312 20.00 ms 12.32 ms -7.68 ms (-38.4%) ok
detect_nested_admin 810 19.97 ms 15.62 ms -4.36 ms (-21.8%) ok
detect_sod 817 19.60 ms 11.49 ms -8.11 ms (-41.4%) ok
stale_grants 827 15.06 ms 9.94 ms -5.12 ms (-34.0%) ok
search_directory 1,147 10.91 ms 8.55 ms -2.36 ms (-21.6%) ok
group_roster 1,047 7.72 ms 7.49 ms -0.23 ms (-3.0%) ok
access_review 801 20.02 ms 11.48 ms -8.54 ms (-42.7%) ok
reporting_path 794 30.98 ms 18.05 ms -12.93 ms (-41.7%) ok
disable_user 644 19.50 ms 11.79 ms -7.71 ms (-39.5%) ok
add_alias 710 19.99 ms 14.28 ms -5.70 ms (-28.5%) ok
revoke_membership 674 8.10 ms 6.37 ms -1.73 ms (-21.4%) ok
detach_device 538 14.79 ms 10.07 ms -4.72 ms (-31.9%) ok
clear_expiry 449 17.39 ms 10.87 ms -6.52 ms (-37.5%) ok

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

Memory shape

Note: loadtest comparison completed with no failing rows.


Memory shape

Status: ok

All memory shapes held. 🟢

base

Given: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling.

When: the base shape is loaded to 0 nodes and 0 relationships, then checkpointed. The child RSS is sampled.

Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries.

Baseline: logical 0.00 MB – 0.00 MB, disk 0.00 MB – 0.00 MB, attributed RSS 0.00 MB – 0.00 MB, ratio 0.000x–0.000x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes.

Validated: logical 0.00 MB, disk 0.00 MB, attributed RSS 0.00 MB, ratio 0.000x.

Report: logical, disk, attributed RSS, and the ratio held.

flat

Given: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling.

When: the flat shape is loaded to 120000 nodes and 0 relationships, then checkpointed. The child RSS is sampled.

Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries.

Baseline: logical 26.02 MB – 26.02 MB, disk 1.95 MB – 1.95 MB, attributed RSS 36.49 MB – 36.93 MB, ratio 1.402x–1.419x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes.

Validated: logical 26.02 MB, disk 1.95 MB, attributed RSS 36.23 MB, ratio 1.392x.

Report: logical, disk, attributed RSS, and the ratio held.

fanout

Given: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling.

When: the fanout shape is loaded to 52016 nodes and 52000 relationships, then checkpointed. The child RSS is sampled.

Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries.

Baseline: logical 28.62 MB – 28.62 MB, disk 1.75 MB – 1.75 MB, attributed RSS 34.93 MB – 36.61 MB, ratio 1.220x–1.279x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes.

Validated: logical 28.62 MB, disk 1.75 MB, attributed RSS 35.96 MB, ratio 1.256x.

Report: logical, disk, attributed RSS, and the ratio held.

ring

Given: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling.

When: the ring shape is loaded to 46000 nodes and 46000 relationships, then checkpointed. The child RSS is sampled.

Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries.

Baseline: logical 29.99 MB – 29.99 MB, disk 1.77 MB – 1.77 MB, attributed RSS 35.30 MB – 35.89 MB, ratio 1.177x–1.197x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes.

Validated: logical 29.99 MB, disk 1.77 MB, attributed RSS 35.18 MB, ratio 1.173x.

Report: logical, disk, attributed RSS, and the ratio held.

dense ring

Given: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling.

When: the dense ring shape is loaded to 5000 nodes and 100000 relationships, then checkpointed. The child RSS is sampled.

Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries.

Baseline: logical 31.41 MB – 31.41 MB, disk 1.62 MB – 1.62 MB, attributed RSS 31.27 MB – 32.46 MB, ratio 0.996x–1.034x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes.

Validated: logical 31.41 MB, disk 1.62 MB, attributed RSS 31.07 MB, ratio 0.989x.

Report: logical, disk, attributed RSS, and the ratio held.

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

Spec-harness coverage — 🟡

  • 🟡 spec coverage 98.28% — 579 genuine failures (1.72% of the executed scenarios)
  • scenarios run: 33725 (passed 33146, failed 579)
  • bands: 🟢 zero failures · 🟡 failure share ≤ 10% · 🔴 beyond (ADR-0482 §4 — the light never fails the build)
  • suite runtime: 1m 43s
  • neo4j_expected_fail (not counted): 327
  • nondeterministic (not counted): 2
  • skipped (not counted): 2323
  • harness ref: 7e21ff7859289596ac3ef3b7f9148f13114e7385
failure category count
error-class-mismatch 241
missing-validation 111
result-mismatch 62
side-effects 51
result-temporal-fidelity 29
syntax-error 28
notification-missing 23
type-error 13
other 10
semantic-error 8
setup-failed-syntax 2
setup-failed-type 1

Rerun spec-harness to refresh. Coverage below 100% never fails the build (ADR-0482 §4).

@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown

Memory controls and workspace memory calibration

Warning: memory controls reported improvement.


Memory controls and workspace memory calibration

Status: improvement

sort — max_rss_to_logical_ratio

Verdict: improvement

Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create.

When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders.

Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive.

Outcome: Expected RSS ratio 1.077x–1.402x. A reading may sit 5% outside that range. Measured 5.31 MB, ratio 0.965x.

Investigate whether this improvement is a false positive. When it is real, re-freeze the envelope with the create-loadtest-baselines workflow.

sort — max_cgroup_to_logical_ratio

Verdict: improvement

Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create.

When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders.

Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive.

Outcome: Expected cgroup ratio 0.999x–1.082x. A reading may sit 5% outside that range. Measured 4.74 MB, ratio 0.862x.

Investigate whether this improvement is a false positive. When it is real, re-freeze the envelope with the create-loadtest-baselines workflow.

expand — max_rss_to_logical_ratio

Verdict: improvement

Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create.

When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders.

Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive.

Outcome: Expected RSS ratio 2.099x–2.147x. A reading may sit 5% outside that range. Measured 35.38 MB, ratio 1.040x.

Investigate whether this improvement is a false positive. When it is real, re-freeze the envelope with the create-loadtest-baselines workflow.

expand — max_cgroup_to_logical_ratio

Verdict: improvement

Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create.

When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders.

Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive.

Outcome: Expected cgroup ratio 2.072x–2.215x. A reading may sit 5% outside that range. Measured 35.49 MB, ratio 1.043x.

Investigate whether this improvement is a false positive. When it is real, re-freeze the envelope with the create-loadtest-baselines workflow.

aggregation — max_rss_to_logical_ratio

Verdict: improvement

Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create.

When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders.

Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive.

Outcome: Expected RSS ratio 1.317x–1.395x. A reading may sit 5% outside that range. Measured 31.54 MB, ratio 1.146x.

Investigate whether this improvement is a false positive. When it is real, re-freeze the envelope with the create-loadtest-baselines workflow.

@oskarhane
oskarhane merged commit d75ef69 into main Nov 9, 2026
77 of 80 checks passed
@oskarhane
oskarhane deleted the fix-upgrade-fixture-versions branch November 9, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant