Repository navigation
refactor(storage): one rule per commit-encode fact, neutral bounded sink (LABS-1292) - #161
Conversation
|
Security review: (security-review skill) Reviewed the WAL/commit encode refactor for reachable security impact: redo-op input validation ( No issues found. Security review: ✅ no issues found |
| fn record(commit_ts: u64) -> AssignedRecord { | ||
| AssignedRecord { | ||
| commit_ts, | ||
| encoded: EncodedOps::encode(&[], &mut WorkspaceCharge::empty()).unwrap(), |
There was a problem hiding this comment.
[Low] Ownership: The new helper encodes with EncodedOps::encode(&[], &mut WorkspaceCharge::empty()), so the temporary charge dies at the end of that statement while the EncodedOps it paid for lives on in the returned AssignedRecord.
Why this matters: EncodedOps::encode documents that the caller keeps the charge for as long as the encoding lives; releasing it early silently drops the buffer's accounting the moment a ledger is installed in this test binary — the exact hole encode_unmetered was added to close in this PR.
Suggested fix: encoded: EncodedOps::encode_unmetered(&[]).unwrap(), and drop the now-unused WorkspaceCharge import.
|
Rust review (rust-reviewer skill, full review) Reviewed the full 14-file diff: the encode-once commit-path refactor in No issues found. Rust review: ✅ no issues found |
| self, | ||
| result: std::result::Result<(), E>, | ||
| serde_error: impl FnOnce(E) -> Error, | ||
| allocation_error: impl FnOnce(usize) -> Error, |
There was a problem hiding this comment.
[Medium] branching (std-2): The terminal now takes a second caller-owned error policy, but only a capture built with a growth policy (capture_charged, which encode_with always uses) can ever record `GrowthFailure::Allocation
Suggested fix: Keep the refusal policy where the sink is built, not at its terminal: pass the allocation renderer into capture_charged (beside Refusal) or render Allocation with the sink's one neutral message, and leave finish(result, serde_error) for the serializer failure only, so no uncharged capture is
finding F-1-1
|
|
||
| fn admits_lsn(self, lsn: u64) -> bool { | ||
| self.lsn.checked_add(1) == Some(lsn) | ||
| } |
There was a problem hiding this comment.
[Medium] branching (q-missing-helper): Tip::next_lsn (60–64) already spells the LSN-advance rule as checked_add(1); admits_lsn re-spells the same relation in the same impl, and the two already disagree on overflow — next_lsn refuse
Suggested fix: Derive one spelling from the other (self.next_lsn().is_ok_and(|next| next == lsn)), or have one method return the successor-or-overflow and let the two enforcement sites render their own message.
finding F-1-2
| result: std::result::Result<(), E>, | ||
| serde_error: impl FnOnce(E) -> Error, | ||
| allocation_error: impl FnOnce(usize) -> Error, | ||
| ) -> Result<u64> { |
There was a problem hiding this comment.
[Medium] boundaries (std-4): The sink resolves one half of its refusal channel from construction-time state (the recorded Refused crossing plus the Refusal the caller configured in capture) but resolves the allocation refus
Suggested fix: Give the growth path the same construction-time policy as Refusal: carry the allocation-refusal rendering beside limit in ChargedGrowth (or let GrowthFailure::Allocation emit its own typed error), so finish stays two-parameter and an uncharged capture supplies nothing for a refusal it ca
finding F-1-3
| /// The rendering the in-flight serializer is aborted with. | ||
| fn abort(&self) -> io::Error { | ||
| match self { | ||
| GrowthFailure::Charge(error) => io::Error::other(error.to_string()), |
There was a problem hiding this comment.
[Medium] boundaries (std-6): The module already owns the canonical renderer for "typed refusal as the in-flight serializer's io abort" — bound_refusal, documented as the one spelling so the abort can never disagree with the ref
Suggested fix: Render the Charge arm through the canonical helper (bound_refusal(error)); keep the allocation arm as the one place that builds a message with no typed error to pass.
finding F-1-4
| fn record(commit_ts: u64) -> AssignedRecord { | ||
| AssignedRecord { | ||
| commit_ts, | ||
| encoded: EncodedOps::encode(&[], &mut WorkspaceCharge::empty()).unwrap(), |
There was a problem hiding this comment.
[Medium] comparison (flag-canonical-dup): This new test re-spells the throwaway encoding the same diff replaced everywhere else: encode_unmetered is documented as the one spelling 'for a throwaway encoding no admission holds a budget for (a
Suggested fix: Use EncodedOps::encode_unmetered(&[]) here as at every other throwaway site, and drop the then-unused WorkspaceCharge import.
finding F-1-5
| .map(|(_, a)| a.commit_ts) | ||
| .map(|(_, a)| a.record.commit_ts) | ||
| .unwrap_or_else(|| shared.graph.commit_ts()); | ||
| last.checked_add(1).map(|next| next.max(slot.txn_id)) |
There was a problem hiding this comment.
[Medium] comparison (std-6): The diff centralized the commit-timestamp validity rule in wal::commit_ts_advances ('neither 0 nor the u64::MAX sentinel', consulted by preflight, append and the scan), but admission re-derives
Suggested fix: Validate the computed commit_ts through the canonical commit_ts_advances (or refuse when next >= u64::MAX) so admission and the WAL agree on the sentinel at every site, and keep the one rule in one place.
finding F-1-6
|
Stellar review (stellar-review skill) Round 3 · head Still open from earlier rounds
Stellar review: |
…ink (LABS-1292) Behavior-preserving structure follow-ups to the encode-once commit path (ADR-0558, ADR-0564): - Accepted no longer carries Option<EncodedOps>; the encoding travels beside it in Admitted and is moved into the append, so the expect is gone. - Wal::preflight_group carries a running tip through a batch, so a batch whose own timestamps do not advance is refused before any frame is written; append_assigned reports each record with its commit_ts. - One commit_ts advance rule and one LSN-successor rule in the WAL, shared by preflight, append and the segment scan; validate_redo_ids lives in the wal module, not behind Wal. - CommitRecord owns the header key spellings and width the encoder stamps. - BoundedSink records one refusal and mints no WAL-named error: the WAL layer words the allocation refusal; the limit narrows to usize once. - EncodedOps::encode_unmetered replaces the throwaway local charges; the charge base is read once. One shared FixedPool test fake. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
cdca440 to
c31736a
Compare
| return Err(error); | ||
| match self.core.refused { | ||
| // Handled above; kept so the match stays exhaustive. | ||
| Some(Refused::Crossed(_)) | None => { |
There was a problem hiding this comment.
[Medium] branching (std-0): The refusal slot is dispatched twice: the early crossed_refusal() matches refused for the crossing, then this match self.core.refused re-examines the same slot and must glue the now-unreachable
Suggested fix: Match self.core.refused once: None maps the serializer's result, Some(Refused::Crossed(at)) returns self.core.refusal.error(at, self.core.limit), and the two Growth arms return their errors — deleting the duplicate dispatch, the merged dead arm, and its comment.
finding F-2-1
| return Err(error); | ||
| match self.core.refused { | ||
| // Handled above; kept so the match stays exhaustive. | ||
| Some(Refused::Crossed(_)) | None => { |
There was a problem hiding this comment.
[Medium] boundaries (std-5): The terminal resolves the recorded refusal twice: crossed_refusal() already matches Refused and returns the crossing error, then the match reads self.core.refused again and folds the already-ret
Suggested fix: Match self.core.refused once in finish: handle Some(Refused::Crossed(crossed_at)) by rendering self.core.refusal.error(crossed_at, self.core.limit) there, keep the growth arms, and let only None take result.map_err(serde_error); delete the early crossed_refusal() call and the dead arm.
finding F-2-2
Oplane Security ReviewOplane identified security requirements relevant to this PR and checked the implementation against them. All requirements are satisfied. Review the generated threat model.
Change comment settings · Set up Oplane in your agent · Workflow guidance |
Spec-harness coverage — 🟡
Rerun |
|
Loadtest baselines comparison
Loadtest baseline comparisonFrozen baselines vs current reports; bands match the Status: ok Pairs: fraud-loadtest: compared · identity-loadtest: compared fraud-loadtest
Per-use-case p99 latency
identity-loadtest
Per-use-case p99 latency
|
|
Memory controls and workspace memory calibration
Memory controls and workspace memory calibrationStatus: mixed sort — max_cgroup_to_logical_ratioVerdict: improvement Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create. When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders. Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive. Outcome: Expected cgroup ratio 0.999x–1.082x. A reading may sit 5% outside that range. Measured 1.25 MB, ratio 0.907x. Investigate whether this improvement is a false positive. When it is real, re-freeze the envelope with the create-loadtest-baselines workflow. expand — max_rss_to_logical_ratioVerdict: improvement Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create. When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders. Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive. Outcome: Expected RSS ratio 2.099x–2.147x. A reading may sit 5% outside that range. Measured 2.15 MB, ratio 1.009x. Investigate whether this improvement is a false positive. When it is real, re-freeze the envelope with the create-loadtest-baselines workflow. expand — max_cgroup_to_logical_ratioVerdict: improvement Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create. When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders. Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive. Outcome: Expected cgroup ratio 2.072x–2.215x. A reading may sit 5% outside that range. Measured 34.00 MB, ratio 1.000x. Investigate whether this improvement is a false positive. When it is real, re-freeze the envelope with the create-loadtest-baselines workflow. aggregation — max_rss_to_logical_ratioVerdict: regression Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create. When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders. Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive. Outcome: Expected RSS ratio 1.317x–1.395x. A reading may sit 5% outside that range. Measured 2.58 MB, ratio 1.497x. This reading is above the frozen envelope. The compare warns and the job stays green until a new envelope is taken. distinct — max_rss_to_logical_ratioVerdict: regression Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create. When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders. Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive. Outcome: Expected RSS ratio 1.256x–1.322x. A reading may sit 5% outside that range. Measured 2.50 MB, ratio 1.451x. This reading is above the frozen envelope. The compare warns and the job stays green until a new envelope is taken. distinct — max_cgroup_to_logical_ratioVerdict: regression Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create. When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders. Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive. Outcome: Expected cgroup ratio 1.098x–1.152x. A reading may sit 5% outside that range. Measured 2.25 MB, ratio 1.306x. This reading is above the frozen envelope. The compare warns and the job stays green until a new envelope is taken. |
|
Memory shape
Memory shapeStatus: ok All memory shapes held. 🟢 baseGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the base shape is loaded to 0 nodes and 0 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 0.00 MB – 0.00 MB, disk 0.00 MB – 0.00 MB, attributed RSS 0.00 MB – 0.00 MB, ratio 0.000x–0.000x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 0.00 MB, disk 0.00 MB, attributed RSS 0.00 MB, ratio 0.000x. Report: logical, disk, attributed RSS, and the ratio held. flatGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the flat shape is loaded to 120000 nodes and 0 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 26.02 MB – 26.02 MB, disk 1.95 MB – 1.95 MB, attributed RSS 36.49 MB – 36.93 MB, ratio 1.402x–1.419x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 26.02 MB, disk 1.26 MB, attributed RSS 37.75 MB, ratio 1.451x. Report: logical, disk, attributed RSS, and the ratio held. fanoutGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the fanout shape is loaded to 52016 nodes and 52000 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 28.62 MB – 28.62 MB, disk 1.75 MB – 1.75 MB, attributed RSS 34.93 MB – 36.61 MB, ratio 1.220x–1.279x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 28.62 MB, disk 1.10 MB, attributed RSS 35.81 MB, ratio 1.251x. Report: logical, disk, attributed RSS, and the ratio held. ringGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the ring shape is loaded to 46000 nodes and 46000 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 29.99 MB – 29.99 MB, disk 1.77 MB – 1.77 MB, attributed RSS 35.30 MB – 35.89 MB, ratio 1.177x–1.197x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 29.99 MB, disk 1.20 MB, attributed RSS 36.20 MB, ratio 1.207x. Report: logical, disk, attributed RSS, and the ratio held. dense ringGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the dense ring shape is loaded to 5000 nodes and 100000 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 31.41 MB – 31.41 MB, disk 1.62 MB – 1.62 MB, attributed RSS 31.27 MB – 32.46 MB, ratio 0.996x–1.034x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 31.41 MB, disk 0.73 MB, attributed RSS 31.72 MB, ratio 1.010x. Report: logical, disk, attributed RSS, and the ratio held. |
Summary
Refactor: structure follow-ups to the encode-once commit path (ADR-0558, ADR-0564) — one rule per fact instead of re-spelled copies, with behavior, error text, figures and byte-identity unchanged. One WAL-group refusal (a batch whose own timestamps do not advance) now fires at preflight, before any frame is written, rather than at append.
Changes
commit.rs:Acceptedno longer holdsOption<EncodedOps>; the encoding travels beside it inAdmittedand is moved into the append, so theexpectis gone.wal.rs/relate-write-local:Wal::preflight_groupcarries a running tip through a batch (F-2-1);append_assignedreports each record with itscommit_tsinstead of zipping positions (F-1-10).wal.rs: onecommit_ts_advancesrule and one LSN-successor rule (Tip) shared by preflight, append and the segment scan (F-1-5, F-1-7);validate_redo_idslives in thewalmodule soencoded.rsno longer importsWal. A non-successor LSN is now refused at preflight with "WAL LSN must advance" (it was previously refused at append as a stale proof; no caller can produce one).relate-engine-api:CommitRecordowns the header key spellings and width bound the encoder stamps, pinned to serde's output by a test (F-1-2).vfs/bounded.rs: the shared sink keeps one refusal slot, mints no WAL-named error (the WAL layer words the allocation refusal), and narrows its bound tousizeonce with a typed error.EncodedOps::encode_unmeteredreplaces the throwawayWorkspaceCharge::empty()encodes and closes the early-release hole; the charge base is read once.relate-test-support::peak_pool::FixedPoolis the one sharedPeakPoolfake (storage tests, alloc test, executor'sPeakPoolDoublealias).append_assignedbatch/commit_ts tests, header-spelling pin.Left out (see LABS-1353): the optional-meter collapse (declined: an empty charge still charges an installed ledger, the hole
encode_unmeteredcloses), nits in untouched code, and thepreflight_withseam, which no longer exists in the tree.Verification
cargo fmt --all --checkcargo clippy --workspace --all-targets --exclude relate-python -- -D warnings(clean)cargo test -p relate-storage -p relate-write-local -p relate-engine-api(incl. payload-cap byte-identity and the alloc-counting memory-bound test)cargo test -p relate --lib(397 passed);cargo test -p relate --no-fail-fast(no failures)stellar-prep.sh local): structure and comparison clean; branching/boundaries nits and one declined design ticketedFixes the structure items of LABS-1292.
🤖 Generated with Claude Code