Repository navigation
perf(executor): sealed+delta adjacency store with watermark compaction (ADR-0551, ADR-0552) - #144
Conversation
|
Security review (security-review skill) Reviewed the sealed+delta adjacency rework (new
Security review: |
| /// in the original chunk order. | ||
| pub(crate) fn decode_chunk(bytes: &[u8]) -> Vec<AdjHop> { | ||
| let n = chunk_len(bytes); | ||
| let mut rest = &bytes[4..]; |
There was a problem hiding this comment.
[Low] Error handling: chunk_len silently reports 0 for a short buffer (None => 0) while decode_chunk immediately slices bytes[4..], panicking on that same input.
Why this matters: the type permits the state — SealedChunk derives Default with empty bytes, so SealedChunk::default().len() is 0 but .decoded() / .visible_hops() panic with a bounds error rather than a codec invariant message, and the two functions disagree about what a short header means.
Suggested fix: read the header once with bytes.get(..4) and return an empty hop stream when absent (or drop the Default derive and make both paths share one expect("codec: chunk header") header read).
| /// LABS-351 histogram maintenance both consume. No string re-interning: | ||
| /// the caller holds the interned type id. | ||
| pub(crate) fn out_chunk_len(&self, node: NodeId, type_id: u32) -> u64 { | ||
| let region = self.sealed_out.get(&node); |
There was a problem hiding this comment.
[High] Concurrency: sealed_out is read before out_adj, inverting the store's lock order.
Why this matters: every other path takes the delta guard first and the sealed guard second (adj_tombstone_deleted, adj_reposition, adj_remove, and compact_side/GC), so a thread here can hold the sealed_out read guard while waiting on out_adj while a concurrent delete/reposition/checkpoint holds the out_adj write guard and waits on sealed_out — an ABBA deadlock on the two IdVec RwLocks (reachable through the executor's public Graph API; relate currently serializes applies, so it is latent rather than impossible).
Suggested fix: take the delta guard first, e.g. let adj = self.out_adj.get(&node); let region = self.sealed_out.get(&node);, then call total_len on the borrowed pair.
| // later hop on its own). | ||
| match sealed | ||
| .and_then(|s| s.chunk(type_id)) | ||
| .filter(|chunk| chunk.len() > 0) |
There was a problem hiding this comment.
[Medium] Memory lifecycle: the len() > 0 filter makes the reclaimable-tombstone gate unreachable for zero-hop chunks, so their tombstones are never reclaimed.
Why this matters: a delete/reposition on a never-sealed chunk records its tombstone on an on-demand zero-hop sealed chunk; with the delta below COMPACTION_DELTA_FLOOR the chunk is never due, so a low-degree (node, type) chunk that keeps one live hop while its other rels are deleted/re-added accumulates one charged tombstone per dead rel without bound (ADR-0551's "tombstone reclaimable at the horizon" gate is skipped for exactly these chunks).
Suggested fix: for a zero-hop chunk, run the fold when chunk.reclaimable(horizon) (the fold drops tombstones whose hop no longer exists) instead of falling through to the floor gate, or drop tombstones whose id has neither a sealed nor a live delta hop.
|
Rust review (rust-reviewer skill, full review) Full rust-reviewer pass over the sealed+delta adjacency rework: the new
Rust review: |
| relate_ir::Direction::Both => { | ||
| len(&self.out_adj, &self.sealed_out) + len(&self.in_adj, &self.sealed_in) | ||
| } | ||
| } |
There was a problem hiding this comment.
[High] branching (std-0): The diff threads a second parallel map through every adjacency accessor, so each of the eleven functions that now takes (map, sealed) re-derives the out↔sealed_out / in↔sealed_in pairing, and the th
Suggested fix: Model one direction's adjacency once: struct AdjSide<'a> { delta: &'a IdVec<TypedAdj>, sealed: &'a IdVec<SealedAdj> } with len/flatten/for_each_hop on it, plus Graph::sides(dir) -> impl Iterator<Item = AdjSide<'_>> yielding one side, or both for Both. The accessors then take a side, `Bot
finding F-1-1
| /// chunks never folded has no entry there, and the checkpoint fold lands one | ||
| /// where it seals. | ||
| #[test] | ||
| fn sealed_side_table_entry_exists_only_where_a_fold_lands() { |
There was a problem hiding this comment.
[Medium] structure (std-1): The file's own module doc scopes it to the adjacency-walk suites (LABS-801), yet this PR appends ~402 lines of a different concern — sealed-tier layout, tombstone, GC and charge-accounting tests — tak
Suggested fix: Move the sealed-tier tests (new lines 371–771) into a sibling graph/tests/adjacency_sealed.rs registered in graph/tests/mod.rs, the same carve-out precedent the sealed proptest file already follows; adjacency.rs keeps only the walk suites its module doc describes.
finding F-1-10
| // A sealed position is superseded at `xmax` (ADR-0551); the | ||
| // new hop lands in the delta. | ||
| let adj_before = adj_floor(&self.out_adj, &self.sealed_out, start) | ||
| + adj_floor(&self.in_adj, &self.sealed_in, end); |
There was a problem hiding this comment.
[Medium] structure (flag-busy-function): A 29-line adjacency reposition-and-charge special case is wedged between put_rel and rel_index_add in the already multi-step property-edit seam, so set_property_rel now interleaves property muta
Suggested fix: Extract the whole if old_ts != new_ts block into a reposition_rel_adjacency(&self, start, end, type_id, id, new_ts, xmax) helper owning both adj_reposition calls and the floor-delta charge, so the property-edit seam reads linearly and the one adjacency-reposition policy has one home.
finding F-1-11
| insert_hop(v, AdjHop { ts, id: rid }); | ||
| } else { | ||
| self.insert(type_id, rid, ts); | ||
| } |
There was a problem hiding this comment.
[Medium] branching (std-0): reposition re-implements insert's existing-vs-create split: the if arm is exactly insert's chunk_mut -> insert_hop path, and the else arm already calls insert. The comment says the ins
Suggested fix: Delete the branch and call the canonical helper unconditionally: self.remove(Some(sealed), type_id, rid); self.insert(type_id, rid, ts);.
finding F-1-12
| // ts 0 hides nothing: a tombstone's xmax is always ≥ 1, so this is | ||
| // the raw physical adjacency in merged order. | ||
| let mut ids = Vec::new(); | ||
| self.for_each_hop(sealed, 0, |hop| ids.push(hop.id)); |
There was a problem hiding this comment.
[Medium] branching (std-2): flatten encodes "no visibility filter" as the magic read timestamp 0 and depends on tombstone stamps always being >= 1, an invariant the adjacency type does not enforce (its own tests pass xmax 0).
Suggested fix: Give flatten a raw read path that takes no read timestamp: merge the delta with SealedChunk::decoded() (or let the visit take Option<u64> where None means unfiltered) so the no-filter mode is explicit instead of a sentinel value.
finding F-1-13
| for _ in 0..NODE_POOL { | ||
| nodes.push(g.create_node(vec!["N".into()], BTreeMap::new()).unwrap()); | ||
| } | ||
| g.commit_write(); |
There was a problem hiding this comment.
[Medium] comparison (flag-copy-paste): This proptest copies the full case harness of adjacency_matches_reference_model (lines 371-387) — the same Graph/NODE_POOL seeding, the same type_ids interning, the same ModelAdj+live initiali
Suggested fix: Extract one fresh_case()/drive(steps) helper in this module that seeds the graph, interns TYPES, and returns/feeds (Graph, nodes, model, live, type_ids); have both proptests call it, the confluence one only adding the fold step.
finding F-1-64
| Some(head) => u32::from_le_bytes(head.try_into().expect("four bytes")) as usize, | ||
| None => 0, | ||
| } | ||
| } |
There was a problem hiding this comment.
[Medium] comparison (flag-silent-fallback): The same 4-byte header is handled two ways: decode_chunk (line 97, &bytes[4..]) and every other codec failure refuse a short stream with a labeled expect, while chunk_len silently answers 0 ho
Suggested fix: Give the header one policy: either expect it in chunk_len (matching the codec's other expects) or return Option<usize> and make SealedChunk always hold the 4-byte empty encoding, so len() and decoded() cannot disagree.
finding F-1-65
| //! | ||
| //! Not to be confused with [`crate::graph::adjacency`], the graph's | ||
| //! adjacency accessors (degree, neighbor iteration, snapshot-visible | ||
| //! traversal) that read and write this tier. |
There was a problem hiding this comment.
[Medium] structure (std-6): The branch adds a second adjacency module beside the existing graph::adjacency, and the confusion is already concrete: this module and graph/adjacency.rs both open with a "not to be confused with"
Suggested fix: Give the new tier a name that says which half it is (e.g. adjacency_store for the chunks, codec and fold) or nest it as graph::adjacency::store; then delete both disambiguation notes and the doubled adjacency import in graph/mod.rs.
finding F-1-7
| bytes: encode_chunk(&[]), | ||
| tombstones: Vec::new(), | ||
| }; | ||
| vec_heap(1, size_of::<(u32, SealedChunk)>() as u64) |
There was a problem hiding this comment.
[Medium] structure (flag-canonical-dup): The admission estimate hand-assembles the region a delete creates instead of asking the code path that creates it: the three terms must mirror SealedAdj::chunk_or_insert (inline empty encoding), `Se
Suggested fix: Derive the term from the real path — add a SealedChunk::empty() used by both chunk_or_insert and this estimate, or build the region (SealedAdj::default(), chunk_or_insert(0).tombstone(0, 1, Supersede::Deleted)) and return its floor_bytes() — so one fact, not two, describes the fresh region
finding F-1-8
| } | ||
| true | ||
| }); | ||
| } |
There was a problem hiding this comment.
[Medium] structure (flag-missed-decomposition): This branch doubles the adjacency state into four parallel tables (out_adj/in_adj plus sealed_out/sealed_in) and then threads the delta/sealed pair by hand through every helper and accessor (flatten_a
Suggested fix: Put the per-direction pair behind one store-tier handle — struct AdjSide { delta: IdVec<TypedAdj>, sealed: IdVec<SealedAdj> } in crate::adjacency, with Graph holding out/in and the compaction cleanup running once per side — so the accessors and free helpers take &AdjSide/&mut AdjSide;
finding F-1-9
|
Stellar review (stellar-review skill) Round 4 · head New findings
Still open from earlier rounds
Stellar review: |
1eaaffe to
492ca81
Compare
Spec-harness coverage — 🟡
Rerun |
| tombstones: Vec<Tombstone>, | ||
| horizon: u64, | ||
| ) -> Vec<Tombstone> { | ||
| let mut sealed_dead: HashSet<RelId> = HashSet::new(); |
There was a problem hiding this comment.
[Low] Consistency: reclaim builds two std::collections::HashSet<RelId> for id membership, while fold_chunk (adjacency/delta.rs) builds the same kind of RelId set as crate::UsedSet (FxHashSet<RelId>) in the same fold pass.
Why this matters: two spellings of one id-set policy sit one call apart, so a later reader cannot tell whether the std set is deliberate; the crate keeps UsedSet at the root for exactly this (ADR-0355 / LABS-337), and the fold pays SipHash per tombstone on the checkpoint path.
Suggested fix:
let mut sealed_dead = crate::UsedSet::default();
let mut deleted_dead = crate::UsedSet::default();and drop the now-unused std::collections::HashSet import in this module.
|
Loadtest baselines comparison
Loadtest baseline comparisonFrozen baselines vs current reports; bands match the Status: ok Pairs: fraud-loadtest: compared · identity-loadtest: compared fraud-loadtest
Per-use-case p99 latency
identity-loadtest
Per-use-case p99 latency
|
|
Memory controls and workspace memory calibration
Memory controls and workspace memory calibrationStatus: mixed sort — max_cgroup_to_logical_ratioVerdict: improvement Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create. When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders. Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive. Outcome: Expected cgroup ratio 0.999x–1.082x. A reading may sit 5% outside that range. Measured 19.24 MB, ratio 0.874x. Investigate whether this improvement is a false positive. When it is real, re-freeze the envelope with the create-loadtest-baselines workflow. expand — max_rss_to_logical_ratioVerdict: improvement Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create. When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders. Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive. Outcome: Expected RSS ratio 2.099x–2.147x. A reading may sit 5% outside that range. Measured 2.14 MB, ratio 1.004x. Investigate whether this improvement is a false positive. When it is real, re-freeze the envelope with the create-loadtest-baselines workflow. expand — max_cgroup_to_logical_ratioVerdict: improvement Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create. When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders. Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive. Outcome: Expected cgroup ratio 2.072x–2.215x. A reading may sit 5% outside that range. Measured 34.00 MB, ratio 1.000x. Investigate whether this improvement is a false positive. When it is real, re-freeze the envelope with the create-loadtest-baselines workflow. aggregation — max_rss_to_logical_ratioVerdict: regression Given: a fresh server per observation, admission on, a hold of 2500 ms, and 1 repeats. Families: sort, expand, aggregation, distinct, and create. When: each observation parks its sessions on RETURN 1, then holds one workspace family at the target reservation. The shapes are one holder at 1.00 MB, 4.00 MB, and 16.00 MB, then the same 16.00 MB across two and four holders. Then: each observation records VmRSS and cgroup deltas against the live held logical bytes. The child stays alive. Outcome: Expected RSS ratio 1.317x–1.395x. A reading may sit 5% outside that range. Measured 2.69 MB, ratio 1.562x. This reading is above the frozen envelope. The compare warns and the job stays green until a new envelope is taken. |
|
Memory shape
Memory shapeStatus: ok All memory shapes held. 🟢 baseGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the base shape is loaded to 0 nodes and 0 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 0.00 MB – 0.00 MB, disk 0.00 MB – 0.00 MB, attributed RSS 0.00 MB – 0.00 MB, ratio 0.000x–0.000x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 0.00 MB, disk 0.00 MB, attributed RSS 0.00 MB, ratio 0.000x. Report: logical, disk, attributed RSS, and the ratio held. flatGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the flat shape is loaded to 120000 nodes and 0 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 26.02 MB – 26.02 MB, disk 1.95 MB – 1.95 MB, attributed RSS 36.49 MB – 36.93 MB, ratio 1.402x–1.419x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 26.02 MB, disk 1.26 MB, attributed RSS 37.16 MB, ratio 1.428x. Report: logical, disk, attributed RSS, and the ratio held. fanoutGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the fanout shape is loaded to 52016 nodes and 52000 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 28.62 MB – 28.62 MB, disk 1.75 MB – 1.75 MB, attributed RSS 34.93 MB – 36.61 MB, ratio 1.220x–1.279x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 27.43 MB, disk 1.10 MB, attributed RSS 35.67 MB, ratio 1.300x. Report: logical, disk, attributed RSS, and the ratio held. ringGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the ring shape is loaded to 46000 nodes and 46000 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 29.99 MB – 29.99 MB, disk 1.77 MB – 1.77 MB, attributed RSS 35.30 MB – 35.89 MB, ratio 1.177x–1.197x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 29.99 MB, disk 1.20 MB, attributed RSS 36.14 MB, ratio 1.205x. Report: logical, disk, attributed RSS, and the ratio held. dense ringGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the dense ring shape is loaded to 5000 nodes and 100000 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 31.41 MB – 31.41 MB, disk 1.62 MB – 1.62 MB, attributed RSS 31.27 MB – 32.46 MB, ratio 0.996x–1.034x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 31.11 MB, disk 0.73 MB, attributed RSS 31.70 MB, ratio 1.019x. Report: logical, disk, attributed RSS, and the ratio held. |
| out.push(width as u8); | ||
| let mut w = BitWriter::new(); | ||
| for &delta in block { | ||
| w.write(delta, width); |
There was a problem hiding this comment.
[Medium] branching (std-0): The codec's block framing — width byte, BitWriter, packed-length arithmetic — is spelled four times: this id-delta loop, the ts-gap loop directly above it, and the two mirrored while remaining > 0 l
Suggested fix: Give the module one block-stream pair — encode_blocks(out, deltas, block) and decode_blocks(rest, count, block, apply) (ts: prev.wrapping_sub(raw); id: prev.wrapping_add(unzigzag(raw))) — and let encode_chunk/decode_chunk pass only the two transforms. The four loops and the duplicated wi
finding F-2-1
| .map(|(_, delta, sealed)| (delta + sealed) as u64) | ||
| .sum::<u64>(); | ||
| c.heap_bytes += adj.floor_bytes(); | ||
| c.heap_bytes += adj.floor_bytes() + region.as_ref().map_or(0, |s| s.floor_bytes()); |
There was a problem hiding this comment.
[Medium] boundaries (std-6): heap_bytes re-spells the per-node adjacency floor that graph::adj_floor (graph/mod.rs:1419) already owns — map.get(node).map(floor) + sealed.get(node).map(floor) — and it cannot call that helper
Suggested fix: Give the store tier the combined measure — TypedAdj::floor_bytes_with(sealed: Option<&SealedAdj>) beside floor_bytes/chunk_lens — and have graph::adj_floor and AdjacencyCensus::of both call it, so the floor rule (delta plus sealed half) has one implementation.
finding F-2-13
| let region = sealed.get(&id); | ||
| c.nodes += 1; | ||
| c.chunks += adj.chunks.len() as u64; | ||
| c.sealed_chunks += region.as_ref().map_or(0, |s| s.chunks.len()) as u64; |
There was a problem hiding this comment.
[Medium] boundaries (std-5): Every sealed read is sealed.get(&id) for an id the delta map yielded, so the census never enumerates the sealed tier: nodes, sealed_chunks, hops and heap_bytes silently omit a region with no
Suggested fix: Walk both stores as first-class sources — a sealed.for_each pass (or a per-direction handle that owns the pair) adding the sealed nodes/chunks/hops/floor bytes — instead of deriving the sealed counts from the delta map's keys; then census_counts_hops_across_the_fold can pin that the two walks ag
finding F-2-14
| adj.remove(region.as_deref_mut(), type_id, id); | ||
| } | ||
| map.remove_if(&node, |adj| adj.chunks.is_empty()); | ||
| sealed.remove_if(&node, |region| region.chunks.is_empty()); |
There was a problem hiding this comment.
[Medium] boundaries (std-7): The delta-entry removal and the sealed-entry removal are two separate store operations after the pair's guards drop at the closing brace: a concurrent adj_tombstone_deleted can see the still-present
Suggested fix: Resolve the pair removal in one critical section — the per-side store handle (or a paired remove_if taking both guards) removes the delta and sealed entries atomically, so a tombstone cannot land in a region whose delta slot is being removed.
finding F-2-15
| (id, expands) | ||
| }) | ||
| .collect() | ||
| }; |
There was a problem hiding this comment.
[Medium] boundaries (std-6): The new test defines a local observable snapshot that re-implements this file's adjacency() helper (lines 217-229) exactly minus the two degree columns, so the one "three-direction neighbors_at
Suggested fix: Delete the closure and compare adjacency(&opened) with adjacency(&writer); if the degree columns are noise, expose an expansions-only projection of adjacency()/NodeAdjacency and use it from both tests.
finding F-2-16
| // position any more — a hop the chain GC already pruned — and the | ||
| // region goes with the last one. The delta hops stay as they are. | ||
| if hops.is_empty() && delta.len() < COMPACTION_DELTA_FLOOR { | ||
| let delta_ids: UsedSet = delta.iter().map(|h| h.id).collect(); |
There was a problem hiding this comment.
[Medium] branching (q-missing-model): The "a never-sealed chunk seals only at the floor" watermark is the due gate's fact (chunk_due: _ => delta >= COMPACTION_DELTA_FLOOR), and the fold re-derives it here with inverted sense to decide
Suggested fix: Let the gate own the verdict: pass chunk_due's decision — or a small value that also names the reason (floor, ratio, reclaimable) — into the fold, and have the zero-hop arm consult that instead of re-comparing delta.len() against COMPACTION_DELTA_FLOOR.
finding F-2-2
| match chunks.iter_mut().find(|(t, _)| *t == type_id) { | ||
| Some((_, hops)) => model_insert_hop(hops, (ts, id)), | ||
| None => chunks.push((type_id, vec![(ts, id)])), | ||
| } |
There was a problem hiding this comment.
[Medium] branching (std-0): ModelAdj::insert already owns the find-or-push chunk insert, yet reposition spells the same match a second time after its removal, so the model's one hop-insert policy lives in two places — the pr
Suggested fix: After the remove-and-drop-emptied block (its chunks borrow has ended), call self.insert(node, out, type_id, id, ts); the duplicated match and its model_insert_hop call site disappear, and the insert path stays the single spelling of find-or-create.
finding F-2-6
| return; | ||
| } | ||
| live.sort_unstable(); | ||
| let id = live[(slot as usize) % live.len()]; |
There was a problem hiding this comment.
[Medium] branching (q-missing-helper): The Delete arm repeats this exact empty-guard, sort and modulo at 347–352, so the strategy's live-slot selection is maintained twice inside one match and a third slot-indexing op would copy it again
Suggested fix: Extract one helper fn live_slot(live: &mut Vec<RelId>, slot: u16) -> Option<usize> that returns None when empty, sorts, and yields slot as usize % live.len(); both arms open with let Some(idx) = live_slot(live, slot) else { return; };, and Delete then takes live.remove(idx) while SetTs
finding F-2-7
| /// The hybrid-codec hop stream, in `cmp_ts_desc` order. | ||
| pub bytes: Vec<u8>, | ||
| pub tombstones: Vec<Tombstone>, | ||
| } |
There was a problem hiding this comment.
[Medium] boundaries (std-5): These two fields are the struct's entire invariant — bytes must always be an encode_chunk stream, and the tombstone set is written only through tombstone — yet both are pub, so any module can
Suggested fix: Make the fields private behind the module's own constructors (SealedChunk::new(hops, tombstones) for the encoding path and a shared SealedChunk::empty()), plus a read-only tombstones() for the assertions; the encoder-validity invariant then has one owner, as with the container-level leaks F-1-
finding F-2-8
| sealed.map_or(0, |s| s.chunk_len(*type_id)), | ||
| ) | ||
| }) | ||
| } |
There was a problem hiding this comment.
[Medium] boundaries (std-5): The accessor answers a positional (u32, usize, usize) whose two counts are interchangeable usizes, and every production consumer immediately re-derives the one number it wants by summing the last
Suggested fix: Give the iterator the per-chunk degree the callers ask for — impl Iterator<Item = (u32, usize)> computing delta.len() + sealed.chunk_len(type_id) once — or, if the split must stay, return a named ChunkLens { type_id, delta, sealed }; the three callers then drop their `|(_, delta, sealed)| delt
finding F-2-9
492ca81 to
ac105d3
Compare
| } | ||
|
|
||
| fn compact_side(&self, delta: &IdVec<TypedAdj>, sealed: &IdVec<SealedAdj>, horizon: u64) { | ||
| delta.retain(|node, adj| { |
There was a problem hiding this comment.
[Medium] Concurrency: compact_side runs the whole fold inside IdVec::retain, which holds the map's single RwLock<Store<T>> write guard while every due chunk is decoded (SealedChunk::decoded) and re-encoded (put_chunk → encode_chunk).
Why this matters: gc_versions runs on the checkpoint timer with readers live, so one checkpoint blocks every concurrent adjacency read (try_for_each_hop_resolved_at, degree_out, …) for the sum of all folds; the rest of gc_versions deliberately re-enters the store one entry at a time via keys().
Suggested fix: mirror the per-id GC pattern — let nodes = delta.keys(); then delta.get_mut(&node) and fold that node, so the map guard is held per node instead of for the whole map.
| let i = match self.chunks.iter().position(|(t, _)| *t == type_id) { | ||
| Some(i) => i, | ||
| None => { | ||
| self.chunks.push(( |
There was a problem hiding this comment.
[Medium] Memory accounting: these pushes into self.chunks (and the one in put_chunk, line ~222) never call fit_reserved, so the Vec keeps Rust's minimum reservation — 4 strides of a 56-byte (u32, SealedChunk) — while SealedAdj::floor_bytes charges vec_heap(len, stride) = one stride for a one-chunk region. The same holds for the tombstones vector put_chunk installs from a collect()ed kept.
Why this matters: vec_heap models capacity as reserved_len(len) and TypedAdj::insert / SealedChunk::tombstone keep that true with fit_reserved (the crate pins capacity() == reserved_len(len) in graph/tests/mod.rs), so the sealed tier here under-charges real heap by up to ~3 strides per region — the exact floor structure_bytes and the heap census report.
Suggested fix: crate::memory::fit_reserved(&mut self.chunks) after both pushes, and fit the incoming tombstones before storing it.
| let mut hops = codec::decode_chunk(&self.bytes); | ||
| if !self.tombstones.is_empty() { | ||
| hops.retain(|h| !self.hidden_at(h.id, ts)); | ||
| } |
There was a problem hiding this comment.
[Medium] branching (q-missing-model): Once a chunk carries a tombstone, every adjacency read of it filters through this retain, and hidden_at (89-91) re-scans the whole tombstone vector per hop, so a chunk with H hops and T tombstones c
Suggested fix: Give SealedChunk one tombstone-visibility lookup: collect the ids hidden at ts once (self.tombstones.iter().filter(|t| t.xmax <= ts).map(|t| t.id)) and filter the decoded stream against that set (or keep the tombstones id-sorted and binary-search), with hidden_at/visible_hops expressed on
finding F-3-1
| let Some(ci) = self.chunks.iter().position(|(t, _)| *t == type_id) else { | ||
| return; | ||
| }; | ||
| let v = &mut self.chunks[ci].1; |
There was a problem hiding this comment.
[Medium] boundaries (std-6): The type-id → chunk-slot lookup has no single owner: chunk spells iter().find, chunk_mut spells iter_mut().find (its doc claims to be "the one find-chunk spelling" the hop policies route throu
Suggested fix: Give each typed-chunk table one indexed lookup (fn chunk_index(&self, type_id: u32) -> Option<usize>) that chunk, chunk_mut, remove and fold_due all route through, and replace SealedAdj::chunk/chunk_mut/chunk_or_insert's find/iter_mut/position trio with the same helper, so the key or
finding F-3-10
| // empty region — and its slot array — behind for nothing. | ||
| if sealed | ||
| .get(&node) | ||
| .is_some_and(|region| region.chunk(type_id).is_some()) |
There was a problem hiding this comment.
[Medium] boundaries (std-5): SealedAdj treats a zero-hop chunk as a first-class state (chunk_or_insert mints one for a delta-only delete, adj_remove writes one back, chunk_len reads it as 0), so the presence probe is true
Suggested fix: Ask the positional fact the comment states — region.chunk_len(type_id) > 0 — or push the supersede decision into the store (TypedAdj::reposition taking Supersede/Option<Supersede>, deciding for itself whether a tombstone is worth recording) so adj_reposition is one unconditional call and t
finding F-3-13
| let mut sealed_out_span = self.sealed_out.span(); | ||
| let mut sealed_in_span = self.sealed_in.span(); | ||
| let mut sealed_out_side = HashSet::new(); | ||
| let mut sealed_in_side = HashSet::new(); |
There was a problem hiding this comment.
[Medium] boundaries (std-4): One concept — a side's sealed-tier estimate state — is carried as four parallel locals plus a shared regions set and then re-selected by `if incoming { (&mut sealed_in_span, &mut sealed_in_side) } e
Suggested fix: Give each side one owned value — e.g. struct SideEstimate { span: usize, side: HashSet<NodeId> } (or a row that carries its own &mut accumulators) — and drive both loops from it; the if incoming re-selection and the bool row tag disappear, and the shadowed sealed name is gone.
finding F-3-14
| let Some((start, end, type_id)) = self.rels.get(id).map(|vers| { | ||
| // Every version of one rel id shares its endpoints and type. | ||
| let newest = vers.last(); | ||
| (newest.live.start, newest.live.end, newest.live.type_id) |
There was a problem hiding this comment.
[Medium] boundaries (std-6): The module already owns this derivation: rel_live_identity (mod.rs:1111) is documented as "the rel's LIVE identity — its endpoints and type" and gc_rel_chain spells the same vers.last() + triple
Suggested fix: Call the existing self.rel_live_identity(id) here (it returns exactly the triple, read the write-window way), and expose that accessor to gc_rel_chain (a &Chain<LiveRel> form so it works under the get_mut guard) instead of letting each site re-derive the chain identity.
finding F-3-15
| a, | ||
| b, | ||
| "T".into(), | ||
| BTreeMap::from([("ts".into(), Value::Int(i))]), |
There was a problem hiding this comment.
[Medium] boundaries (std-6): Lines 536-550 hand-roll the exact sealed-floor fixture this PR adds as test_support::folded_floor_chunk — fresh endpoints, 32 ascending-ts :T hops, one write, the fold, then `interner.get_id("T"
Suggested fix: let (a, b, _, type_id) = crate::graph::test_support::folded_floor_chunk(&g); and drop the hand-rolled create loop and fold (keep floor for the delta hops' ts values); the later compact_adjacency_at_watermark(0) stays as the test's own second fold.
finding F-3-16
| for type_id in due { | ||
| if let Some(ci) = self.chunks.iter().position(|(t, _)| t == type_id) { | ||
| changed |= self.fold_chunk(sealed, ci, *type_id, horizon); | ||
| } |
There was a problem hiding this comment.
[Medium] branching (flag-silent-fallback): A due id the scan cannot find is silently skipped, so a stale or duplicated entry would leave a chunk unfolded with no error — the silent-drop shape the code-shape rules ban. Today due_chunks yields
Suggested fix: Assert the invariant (let ci = self.chunks.iter().position(...).expect("a due chunk still exists")) or fold in one pass over self.chunks with due-ness decided per entry so no id lookup can miss; do not keep a branch whose only outcome is a silently skipped fold.
finding F-3-2
| if sealed | ||
| .get(&node) | ||
| .is_some_and(|region| region.chunk(type_id).is_some()) | ||
| { |
There was a problem hiding this comment.
[Medium] branching (std-0): This new guard splits adj_reposition into two modes chosen by probing the sealed tier's per-type chunk table from the graph layer, and the else arm hand-replays the delta-only half of `TypedAdj::r
Suggested fix: Move the mode into the store tier the way adj_remove already takes its sealed side: TypedAdj::reposition(sealed: Option<&mut SealedAdj>, …) where the tombstone is recorded only for an existing chunk_mut(type_id) (SealedAdj::supersede, one documented policy sitting beside chunk_or_insert),
finding F-3-7
| (&mut sealed_in_span, &mut sealed_in_side) | ||
| } else { | ||
| (&mut sealed_out_span, &mut sealed_out_side) | ||
| }; |
There was a problem hiding this comment.
[Medium] branching (q-missing-model): The slot-array charge is one three-part protocol — a span, a first-seen node set, and the store's slot/overflow strides — but it is held as bare locals: the create arm already hand-pairs `out_sp
Suggested fix: Extract the protocol once — SlotSpan { span: u64, seen: HashSet<NodeId>, slot: u64, overflow: u64 } with grow(&mut self, node) -> i64 — and instantiate one per store (SlotSpan::of(&self.out_adj), the sealed pair likewise), so each arm carries [SlotSpan; 2], iterates its two rows, and the `if
finding F-3-8
| match self.chunk_mut(type_id) { | ||
| Some(slot) => *slot = chunk, | ||
| None => self.chunks.push((type_id, chunk)), | ||
| } |
There was a problem hiding this comment.
[Medium] boundaries (std-6): SealedAdj::chunks is the one adjacency container grown by push without the crate's capacity helper (chunk_or_insert pushes the same way at 194-202), while SealedAdj::floor_bytes prices it with
Suggested fix: Call crate::memory::fit_reserved(&mut self.chunks) after the None-arm push in both put_chunk and chunk_or_insert (or route both growths through one insert helper that fits), so the table's capacity is the reserved_len(len) its floor_bytes charges.
finding F-3-9
…n (ADR-0551, ADR-0552) Committed adjacency becomes a sealed hybrid-codec region plus a committed delta per (node, rel type, direction) chunk, with an in-flight overlay, so a checkpoint folds hot deltas into a packed, compressed region instead of keeping every hop as a live sorted entry. - new private `adjacency` module: bitpack + hybrid chunk codec, the `SealedAdj`/`SealedChunk` tier, `TypedAdj` delta, tombstones with separate supersede and delete version stamps, watermark compaction (delta > 25% of sealed, never-sealed delta at the floor, reclaimable tombstone at the GC horizon) driven by the existing `gc_versions` pass - reads merge delta-first then sealed in cmp_ts_desc order; deletes of sealed hops record versioned tombstones, so a pinned snapshot keeps its frozen adjacency; a delete in a never-sealed chunk records its tombstone on an on-demand zero-hop chunk rather than letting the fold seal a ghost - the delete tombstone records without a delta-entry precondition: the tombstone lives in the sealed region, so a delete landing in the window `adj_remove` leaves between its two side-table removals still hides the hop; a reposition of a delta-only hop records no tombstone at all, so it no longer leaves an empty region behind - a zero-hop sealed region is due for the fold on a reclaimable tombstone and drops with the last one, so a sub-floor chunk churning delete or reposition no longer accumulates tombstones across checkpoints; the per-node length accessor takes the delta read guard before the sealed one, matching every sibling accessor - the memory floor charges the sealed tier's slot arrays beside each region's floor — the term the heap census reads as `slot_bytes` — and a self-loop's delete charges both sides' on-demand regions - the watermark fold holds the adjacency write guard per node, not across the whole map, so a concurrent reader stalls for one node's fold instead of the checkpoint's; the sealed tier's vectors fit their reservation, so the floor charge is the vectors' real heap - the chunk codec names its header and block-width invariants instead of fabricating a zero length or a zero width over them - the sealed tier is a Graph-level side table keyed by node and direction, so a node that never folds pays nothing: `TypedAdj` stays 24 bytes and the pinned logical-memory shapes are byte-identical to main - every reader (neighbor walks, `incident_rels`, `redo_preflight`, the research neighbor walk, the heap census) goes through the one merged seam - the ADR-0552 8-property proptest reference-model suite pins the observable adjacency before and after the rework, unmodified Verification: `cargo test -p relate-executor` 1444 passed; `--features research` green (incl. the census fold test); `relate-storage` 481 passed; `relate` 1488 passed; `relate-datasets` memory-shape pins unchanged and passing; `cargo fmt --all --check` and the workspace clippy gate clean; `cargo xtask check-boundaries` 0 violations. Known shortfall: the SF0.1 out+in adjacency census lands at 119.0 MiB (1.24x from the 147.4 MiB baseline) rather than the 3x the destination aimed for — the compressed payload falls to 48.7 MiB but the container and slot floor (~69.7 MiB) dominates. The floor restructuring is filed as a follow-up; query-latency (ADR-0245 bands) was not run. Co-Authored-By: Claude Code <noreply@anthropic.com>
ac105d3 to
6fc7154
Compare
Oplane Security ReviewOplane identified security requirements relevant to this PR and checked the implementation against them. All requirements are satisfied. Review the generated threat model.
Change comment settings · Set up Oplane in your agent · Workflow guidance |
| /// reposition keeps the sealed stamp and adds the delete stamp: the | ||
| /// sealed position was superseded at the reposition, the relocated delta | ||
| /// hop dies at the delete. | ||
| pub(crate) fn tombstone(&mut self, id: RelId, ts: u64, supersede: Supersede) { |
There was a problem hiding this comment.
[Medium] Denial of service: tombstone bookkeeping is a linear scan on both the write and the read path, so bulk deletes and repeated reads of a tombstoned chunk scale quadratically.
Why this matters: tombstone scans every existing tombstone per delete (iter_mut().find), so K deletes on one (node, type, direction) chunk cost O(K²) — one MATCH (n)-[r:T]->() DELETE r over a hub pays that while the write runs — and visible_hops/hidden_at (lines 89–90, 127–130) scans all K tombstones for every one of N hops, costing O(N·K) per neighbor expansion whenever the tombstones are not yet reclaimable (a pinned snapshot, or the window before the next checkpoint).
Suggested fix: key tombstones by id (a HashMap<RelId, Tombstone> beside the Vec, or keep it sorted and binary-search), and build one hidden-id HashSet per chunk per read instead of scanning the tombstone list per hop.
| }; | ||
| delta += span_growth(span, node, sealed_slot, sealed_overflow, side); | ||
| } | ||
| delta += crate::adjacency::fresh_sealed_region_bytes() as i64; |
There was a problem hiding this comment.
[Medium] Resource exhaustion: the new per-delete sealed-region estimate is refusable, so a relationship DELETE at the logical memory cap fails instead of committing.
Why this matters: this line adds fresh_sealed_region_bytes() to RedoDelta::total but not to tombstone_spill, so admit_redo_capacity sees a positive refusable delta on a delete and returns Error::Capacity; ADR-0529 and logical_graph_size_limit_delete_then_create_recovers require a delete at the cap to commit so space can be reclaimed, and the rel-chain spill is already exempt for exactly this reason.
Suggested fix: report the sealed-region growth as part of the delete's tombstone spill (thread it out of estimate_adj_delta into RedoDelta::tombstone_spill) so refusable subtracts it, matching the rel-chain spill.
| let sealed = region | ||
| .as_ref() | ||
| .is_some_and(|region| region.chunk(type_id).is_some()); | ||
| if !sealed && regions.insert((incoming, node, type_id)) { |
There was a problem hiding this comment.
[Medium] Memory accounting: estimate_adj_delta prices a delete's on-demand sealed region but not the tombstone that same delete records on an already-sealed chunk.
Why this matters: for a hop whose (node, type) chunk is already sealed, delete_rel still calls adj_tombstone_deleted, which pushes a Tombstone and can re-grow the vector — here sealed == true, so this loop skips the charge entirely. admit_redo_capacity only sees this estimate, so a batch of sealed-hop deletes (or a SET r.ts that repositions one, which has no estimate branch at all) is admitted with none of the bytes it retains and used_bytes can land past logical_graph_size_limit — the failure mode the stats/constraint entry terms exist to close.
Suggested fix: charge one Tombstone per delete when sealed is true (deduped per side/node/type in regions) and cover the reposition path; if deletes must stay unrefusable like the chain spill (ADR-0529), report the same term in tombstone_spill instead of total.
| /// One sealed side-table id slot, the stride of the sealed stores' arrays. | ||
| fn sealed_slot() -> u64 { | ||
| std::mem::size_of::<Option<SealedAdj>>() as u64 | ||
| } |
There was a problem hiding this comment.
[Medium] structure (std-1): graph/mod.rs is the crate's hub module and this PR grows it 1431 -> 1511 lines with the sealed tier's whole pricing family: two more stride helpers here, the sealed terms in structure_bytes, and a sec
Suggested fix: Move the adjacency/sealed pricing and estimate family — adj_slot/adj_value/sealed_slot/sealed_value, hop_counts, adj_floor, span_growth, bump_adj and estimate_adj_delta — into graph/adjacency.rs (which already owns the adjacency accessors and imports hop_counts) or a graph/adjacency_estimate.rs besi
finding F-4-1
| // prices, plus the slot arrays that hold them — the term the heap | ||
| // census reads as `slot_bytes`. | ||
| n = n.saturating_add(self.sealed_out.array_bytes()); | ||
| n = n.saturating_add(self.sealed_in.array_bytes()); |
There was a problem hiding this comment.
[Medium] structure (std-6): structure_bytes states its own composition rule a few lines below this hunk — "Each family's term lives with the family ... and composes here, at the oracle's one site" — and every sibling obeys it (d
Suggested fix: Add adjacency_structure_bytes() beside the adjacency accessors in graph/adjacency.rs (out_adj/in_adj slot arrays + floors, sealed slot arrays + region floors) and call it at the oracle beside the three sibling family terms, so the module that owns the adjacency layout also prices it.
finding F-4-2
Summary
Optimization: committed adjacency becomes a sealed hybrid-codec region plus a committed delta per (node, rel type, direction) chunk, folded at checkpoint by the existing GC pass — so a store stops paying live sorted entries for cold hops. Ports the sealed+delta store designed in ADR-0551/0552 into the engine under the adjacency seam.
Why
Measured on the LDBC SF0.1 store with the heap-census harness: the out+in adjacency census was 147.4 MiB, of which the per-hop live entries dominated. This PR cuts the compressed payload of that term from 85.6 MiB to 48.7 MiB. The container/slot floor around it (~69.7 MiB) is untouched, so the census lands at 119.0 MiB total — 1.24x from the baseline, not the 3x the destination aimed for. That gap is the floor, not the codec, and is filed as a follow-up (see below).
Changes
relate-executor/src/adjacency/: the new private store tier — bitpack and hybrid chunk codec,SealedAdj/SealedChunk, theTypedAdjdelta, versioned tombstones with separate supersede and delete stamps, and watermark compaction (delta > 25% of sealed, never-sealed delta at the floor, reclaimable tombstone at the GC horizon) driven bygc_versionscmp_ts_descorder through one seam (graph/adjacency.rs); every reader — neighbor walks,incident_rels,redo_preflight, the research neighbor walk, the heap census — goes through itadj_removeleaves between its two side-table removals still hides the hop; a reposition of a delta-only hop records no tombstone at all and no longer leaves an empty region behindslot_bytes— and a self-loop's delete charges both sides' on-demand regionsTypedAdjstays 24 bytes and the pinned logical-memory shapes are byte-identical to mainstore.rs941 → 791 lines as the adjacency policy moves to its own moduleVerification
cargo test -p relate-executor— 1444 passed, 3 ignoredcargo test -p relate-executor --features research census— passes (census counts hops across a fold)cargo test -p relate-storage— 481 passedcargo test -p relate— 1488 passedcargo test -p relate-datasets --test memory_shapes— 4 passed, pinned integers byte-identical to maincargo fmt --all --checkandcargo clippy --workspace --all-targets --exclude relate-python -- -D warnings— cleancargo xtask check-boundaries— 42 packages, 0 violationsadj_remove's two side-table removals), the floor's missing sealed slot-array charge, the self-loop's second on-demand region, the codec's fabricated zero length/width, the whole-map write-lock hold across the checkpoint fold, and the sealed vectors' unreserved capacityscripts/premerge-check.sh 144— the merged tree compilesmake query-latency(ADR-0245 bands) — not run; the ready-gate lane is the first to measure itDeferred, with the reason stated on this PR: the stellar rounds reported 73 findings (2 High, 63 Medium, 8 Low) across two heads. Both High findings are the same shape — the
(delta, sealed)pair threaded by hand through the reader and writer seams — which the end-of-feature audit also raised and judged not worth the churn on this branch; that cluster is now a ticketed work item rather than parked polish, and it is the head of the floor restructuring (LABS-1312). The remaining refactor-shaped findings (canonical-helper duplication, parallel state, representation leaks) sit on the review-tasks issue for this PR. Findings the rounds raised on files this PR never touches (exec/…,eval/quantifier.rs,overlay/writes.rs) are pre-existing code pulled in by the review's dependency expansion, left to stellar's own ledger.Fixes LABS-270.