Repository navigation
SURF-1407: Bound user-controlled distance in apoc.neighbors - #975
Merged
JoelBergstrand merged 4 commits intoSep 28, 2026
Merged
JoelBergstrand merged 4 commits into
JoelBergstrand merged 4 commits into
Conversation
…number.exact and apoc.coll These functions fed caller-supplied sizes straight into heap allocations and loop bounds, with no upper bound and no overflow-checked narrowing. Neo4j's transaction memory tracker does not see plain Java objects allocated inside a procedure body, so the query was never aborted and the JVM hit OutOfMemoryError - which, with the recommended -XX:+ExitOnOutOfMemoryError, takes down the whole DBMS process. Each site now estimates its result size with overflow-checked arithmetic and charges it to the transaction memory budget through the preview org.neo4j.procedure.memory.ProcedureMemory API before allocating anything, so dbms.memory.transaction.* decides what is allowed and the cost is visible in SHOW TRANSACTIONS. Degenerate arguments are rejected up front with a message naming the argument. Shared helper: * common/.../apoc/util/ProcedureMemoryUtil - one home for the String size estimate and for the charge-then-allocate sequence. Closes the tracker when a charge is refused, which otherwise leaks it for the rest of the transaction, and saturates rather than overflowing so an unrepresentable size reports its size instead of an arithmetic overflow. Per function: * apoc.text.repeat - short-circuits an empty item, which also removes an infinite, zero-allocation loop that no query timeout or dbms.killQuery could stop: a count above 2^31 wrapped the int counter while still comparing less than the long bound. Replaces the loop with String.repeat, and rejects a negative count, a count no String could hold, and a count whose product with the item length exceeds the maximum string length. * apoc.text.lpad / rpad - rejects an empty delimiter, which used to surface as StringIndexOutOfBoundsException, and builds the result once instead of allocating both a pre-sized StringBuilder and a separate char array, which together asked for roughly twice the final size. * apoc.text.random - rejects a negative length, and an empty character set after the A-Z/a-z/0-9 expansion. * apoc.text.format - the output size is driven by the format string rather than by any argument, so this is the one site that cannot estimate up front. Charges incrementally through TrackedAppendable, which accounts for the buffer as its capacity grows. * apoc.number.exact.add / sub / mul / div - charges the eventual plain-string length before the arithmetic. All four share the sink: 1e1000000000 parses in microseconds but expands to a two-gigabyte string. toInteger and toFloat are deliberately untouched, as neither materialises a plain string. * apoc.coll.combinations - computes the result cardinality with a saturating binomial and charges it before building, so combinations(range(1,40), 20) fails in microseconds rather than after seconds of allocation. Selection sizes outside the int range are now rejected instead of silently truncating: combinations([1,2,3], 4294967298) used to narrow to 2 and return a result. Null arguments to any of these functions are now rejected by name, checked left to right. The numeric arguments move from long to Long so that a null reaches APOC code at all: previously it stopped at the framework's argument coercion and surfaced as "ClassCastException: NoValue cannot be cast to NumberValue", naming internal Neo4j value classes. Both types map to INTEGER, so the published signatures are unchanged. apoc.text.repeat(null, n) therefore reports "'item' must not be null" rather than the NullPointerException it used to throw; there is no null-in-null-out convention in apoc.text to prefer instead, as 21 of its 44 functions return null for a null string argument while 15 throw NullPointerException and 5 return something else. Tests: argument validation alongside the existing cases in StringsTest, CollTest and ExactTest; memory accounting in new Strings/Coll/Exact MemoryTrackingTest classes running with db.memory.transaction.max=4m, so oversized calls are refused before they allocate and are safe to run in CI. The hostile-input cases assert on the error under a preemptive timeout, so a regression reinstating the allocation fails loudly rather than exhausting the test heap. No procedure or function name, signature or description changes, so the generated apoc.ApocSignatures and StartupTest are unaffected.
loveleif
reviewed
Sep 25, 2026
JoelBergstrand
force-pushed
the
surf-1407-user-controlled-distance-drives-giant-allocations-and
branch
from
September 25, 2026 07:58
b209b47 to
24ce312
Compare
JoelBergstrand
force-pushed
the
surf-1407-user-controlled-distance-drives-giant-allocations-and
branch
from
September 25, 2026 08:19
24ce312 to
2d6e747
Compare
apoc.number.exact.div charged the same result estimate as add, sub and mul - both operands' plain lengths plus four - but an exact quotient can be longer than both operands together: 1 / 2^k has k digits while 2^k has only about 0.3 * k. The division estimate now allows up to four result digits per divisor digit (log2(10) rounded up). The new test divides by 2^1400000 under the 4 MiB limit; with the old estimate the charge fits and the division runs until the test times out. Also cache the ArrayList shallow size in apoc.coll.combinations in a static constant. HeapEstimator.shallowSizeOfInstance uses reflection and the value is fixed per JVM.
Problem - the six apoc.neighbors procedures trusted the caller's `distance`: * byhop / athop (and .count) allocated one bitmap per hop up front: distance = 2000000000 asked for gigabytes of untracked heap, and larger values threw NegativeArraySizeException or ArrayIndexOutOfBoundsException. * tohop / tohop.count looped forever on distance = 9223372036854775807, pinning a core until restart; no query timeout or kill could stop it. * byhop streams exactly `distance` rows, so a huge value streamed billions of empty rows. Fix: * Traverse lazily and stop as soon as a hop reaches no new node, so a huge distance on a finite graph costs nothing. * Charge the traversal to the transaction memory budget, so a deep search on a large graph is refused by db.memory.transaction.max instead of filling the heap. * Honour query timeouts and kills during traversal. * New setting apoc.max.hops (default 10000) bounds byhop / byhop.count only, whose output grows with distance rather than with the graph. byhop still returns exactly `distance` rows. * Reject a null `node` or `distance` by name, e.g. "'distance' must not be null". Tests: * NeighborsTest - huge distances return promptly on a path and a cycle; the apoc.max.hops bound; byhop row count; null arguments. * NeighborsMemoryTrackingTest - deep searches fail under a 4 MiB limit and succeed under 256 MiB. * ApocConfigTest / ApocConfigCommandExpansionTest - apoc.max.hops default and rejection of non-positive values. No procedure names or signatures change.
HeapEstimator.shallowSizeOfInstance uses reflection, so cache the fixed per-JVM sizes in static constants instead of computing them per call.
JoelBergstrand
force-pushed
the
surf-1407-user-controlled-distance-drives-giant-allocations-and
branch
from
September 25, 2026 09:48
2d6e747 to
2b64652
Compare
JoelBergstrand
deleted the
surf-1407-user-controlled-distance-drives-giant-allocations-and
branch
September 28, 2026 10:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stands ontop of this PR
Problem - the six apoc.neighbors procedures trusted the caller's
distance:byhop/athop(and.count) allocated one bitmap per hop up front:distance = 2000000000asked for gigabytes of untracked heap, and largervalues threw
NegativeArraySizeExceptionorArrayIndexOutOfBoundsException.tohop/tohop.countlooped forever ondistance = 9223372036854775807,pinning a core until restart; no query timeout or kill could stop it.
distancerows, so a huge value streamed billionsof empty rows.
Fix:
distance on a finite graph costs nothing.
on a large graph is refused by
db.memory.transaction.maxinstead offilling the heap.
apoc.max.hops(default 10000) boundsbyhop/byhop.countonly, whose output grows with distance rather than with the graph.
byhop still returns exactly
distancerows.nodeordistanceby name, e.g."'distance' must not be null".
Tests:
NeighborsTest- huge distances return promptly on a path and a cycle;the
apoc.max.hopsbound; byhop row count; null arguments.NeighborsMemoryTrackingTest- deep searches fail under a 4 MiB limit andsucceed under 256 MiB.
ApocConfigTest/ApocConfigCommandExpansionTest-apoc.max.hopsdefaultand rejection of non-positive values.
Docs PR