Skip to content

feat(executor): fulltext accent folding and wildcard terms (LABS-1460, ADR-0583) - #232

Merged
jexp merged 1 commit into
mainfrom
feat/fulltext-fold-wildcard
Nov 12, 2026
Merged

jexp merged 1 commit into
mainfrom
feat/fulltext-fold-wildcard

Conversation

@jexp

@jexp jexp commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

New feature (its stacked base #227 has merged): a standard-folding fulltext index now folds Latin diacritics, and the query syntax gains wildcard terms (term*, te?m, *term). Reported against the Aura speaker template: eifrem missed Eifrém, andreas sateras missed Andreas Satéras, and a Kollegg* prefix matched nothing.

Changes

  • standard-folding is a real analyzer (ASCII folding of Latin diacritics, ligatures and letter variants; Greek and Cyrillic untouched). The index's configured analyzer is read, so stored tokens and analyzed query terms both fold.
  • Wildcards: * any run, ? one character, leading wildcard, field:term*, */*:* match-all, \* escape, quoted wildcards are plain text. They expand over each field's vocabulary into the existing any-of group; a matching field scores a constant 1.0 times the boost; no expansion cap.
  • Under a folding analyzer the literal parts of a wildcard are folded like stored tokens, so Müll*, mull*, MÜLL* and EIFRÉ* all find Müller/Eifrém. This is a deliberate deviation from live Neo4j 2026.09, which leaves wildcards unfolded (recorded in ADR-0583). ? counts one character of the folded token (stra?e does not match strasse).
  • Fuzzy terms stay lowercased only (live behaviour), so eifrém~0 still misses on a folding index.
  • ADR-0583 (Proposed) records the decision and the live calibration (Neo4j 2026.09-enterprise).
  • docs/reference/cypher-syntax.md fulltext table and folding note updated.

User-facing impact

Folding indexes match accent-insensitively; * and ? outside quotes in a fulltext query now mean wildcard (before they were separators).

Example usage

CREATE FULLTEXT INDEX speakerText FOR (p:Speaker) ON EACH [p.name]
  OPTIONS {indexConfig: {`fulltext.analyzer`: 'standard-folding'}};
CALL db.index.fulltext.queryNodes('speakerText', 'eifrem')    // finds Eifrém
CALL db.index.fulltext.queryNodes('speakerText', 'Kollegg*')  // finds Kollegger, Kolleggers

Verification

  • cargo test -p relate-executor fulltext (42 passed), cargo test -p relate --test fulltext_search_semantics (19 passed; SEARCH clause and procedures)
  • cargo fmt --all --check; rustup run 1.88 cargo clippy --workspace --all-targets --exclude relate-python -- -D warnings
  • Behaviour calibrated against live Neo4j 2026.09-enterprise (2026.10.1 was not available)
  • Local stellar round: no blocker/critical/high; one medium fixed (analyzer lookup moved next to the DDL config key), nits deferred
  • Full matrix CI (draft runs the fast gate only)

Fixes LABS-1460.

🤖 Generated with Claude Code

Rebased onto current main after #227 landed. While rebasing, its head was found to silently revert main's #244 (creation_failure wrapper in the constraint matrix) and #245 (the ddl test's constraint-equivalence token) — its merge-base predated those fixes, so the revert was textually clean. The commit itself touches only its own 10 files, so it was cherry-picked onto current main; set-equality now lists exactly those 10 files with no reverts.

Target::Fuzzy { term, edits } => {
FieldMatch::Weighted(expand_fuzzy(term, *edits, stat))
}
Target::Wildcard(pattern) => FieldMatch::Constant(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] Denial of service: uncapped wildcard expansion is re-materialized per clause and per field, so a short query forces memory proportional to clauses times vocabulary.

Why this matters: every wildcard clause clones each matching vocabulary term into its own HashSet (stat.df.keys()...cloned().collect()), so a query of N repeated * tokens allocates N copies of the index vocabulary; over Bolt/HTTP (16 MiB statement/message limit) a few KiB of * can become gigabytes and OOM the store, aborting every session on it.
Suggested fix: expand each distinct (field, pattern) once per scan and share the set across identical clauses (or test document tokens against the Pattern directly), and add a test that peak memory does not scale with clause count.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Security review (security-review skill)

The diff adds standard-folding accent analysis and */? wildcard terms to the fulltext scanner, shared by the SEARCH clause and the db.index.fulltext.queryNodes/queryRelationships procedures. No auth, secret, or dependency changes; the new glob matcher is iterative (no recursion), the query lexer's slicing is char-boundary safe, and unknown stored analyzer names fail closed as index-definition drift. The one concern is resource exhaustion: wildcard expansion is uncapped and rebuilt per clause, so a user-controlled query can multiply the field vocabulary by clause count and OOM the process.

  • Medium — relate/relate-executor/src/exec/fulltext.rs:440: unbounded wildcard expansion allocates one vocabulary-sized HashSet<String> per field per wildcard clause; a query with many * clauses (4 MiB statement cap or a large procedure parameter) amplifies memory by the clause count and bypasses the ADR-0463 allocation budget and fuzzy's 50-term cap.

Security review: ⚠️ 1 issue(s) flagged inline

fulltext::Analyzer::from_value(&eval(graph, input, expr, params, cypher_version)?)
})
.transpose()?;
let analyzers = fulltext::Analyzers::new(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] Option handling: WITH ANALYZER NULL becomes an explicit Standard override instead of the documented "no override", so it bypasses a folding index's query-side folding.

Why this matters: Analyzer::from_value maps Value::Null to Analyzer::Standard (fulltext.rs:141-148 calls NULL CIP-225's "no override"), and the new Analyzers::new(index, Some(Standard)) path then skips index.query_default(); on a standard-folding index an accented term such as Eifrém misses with WITH ANALYZER NULL but matches when the clause omits it.
Suggested fix: keep NULL as absent — map Value::Null to None before from_value (or return Option<Analyzer> from the override resolution) so Analyzers::new applies query_default(); add a folding-index test for WITH ANALYZER NULL.

for term in tokenize(term) {
let edits = suffix
.fuzzy
.map_or(0, |fuzz| fuzz.edits_for(term.chars().count()));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] Option handling: suffix.fuzzy.map_or(0, ...) fabricates an exact-match default for a value the enclosing else if suffix.fuzzy.is_some() already guarantees is Some.

Why this matters: the None arm is unreachable today, but if that guard changes every fuzzy clause silently degrades to distance 0 instead of failing — the repo bans fabricated defaults over invariant-bound values.
Suggested fix: bind the invariant once (let Some(fuzz) = suffix.fuzzy else { return Err(malformed(text, "a fuzzy suffix is required here")); };) and call fuzz.edits_for(...) in the loop.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Rust review (rust-reviewer skill, full review)

Full-diff review of the executor-side fulltext change: standard-folding accent folding on the index and query sides, wildcard term lexing/parsing/matching over each field's vocabulary, and the analyzer plumbing through both the SEARCH clause and the db.index.fulltext.query* procedures. Ownership, lifetimes, unsafe usage (none) and error handling are sound, and the new paths carry unit plus integration coverage. The two medium findings already posted on this head (explicit WITH ANALYZER NULL silently bypassing the index's query default, and uncapped per-clause wildcard vocabulary cloning) are deferred to LABS-1491 per the maintainer's merge decision; this round surfaced one new Low and no new blocking issue.

  • Low — relate/relate-executor/src/exec/fulltext/fold.rs:47: the fold's ASCII gate is applied before SPECIAL, so precomposed ǣ/ǿ never fold to ae/o.

Rust review: ✅ no issues found

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stellar review round 1: 12 inline finding(s); see summary comment.

pub(crate) fn index_analyzer(
options: &BTreeMap<String, Value>,
index_name: &str,
) -> Result<Analyzer> {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] structure (std-6): This is the fulltext scan's stored-config read — it turns a stored indexConfig value into a query-time Analyzer and raises a query-time drift error — but it lives in the schema-DDL options module,

Suggested fix: Move the stored read beside the analyzer policy (e.g. fulltext::Analyzer::from_index_options(options, index_name)) and give the scan one resolution entry point — resolve_fulltext_index/Analyzers::for_index(options, query_override) — that both exec::search and exec::procedures call, mirrori

finding F-1-1

Error::execution(format!(
"InvalidIndexDefinition: fulltext index {index_name} has an invalid indexConfig option"
))
};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] comparison (flag-canonical-dup): The stored-option drift contract already exists as exec::vector::invalid_index_option (vector.rs:172-176), whose doc claims it is "One drift contract for every invalid stored index option" and whose s

Suggested fix: Hoist one index-kind-aware drift constructor into this shared options module (the layer exec::vector already imports) and call it from both stored_config and index_analyzer; drop the local closure.

finding F-1-10

.as_ref()
.map(|expr| {
fulltext::Analyzer::from_value(&eval(graph, input, expr, params, cypher_version)?)
})

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] comparison (std-5): WITH ANALYZER's two spellings of "no analyzer named" now diverge: an absent clause reaches Analyzers::new as None and follows the index (Analyzer::query_default), while an explicit NULL is lowered to

Suggested fix: Give "no analyzer named" one representation: resolve the evaluated WITH ANALYZER to Option (NULL -> None) if NULL is no-override, or to a named three-state enum if NULL must mean standard; fix the comments to state the chosen policy and pin it with a WITH ANALYZER null case on a folding in

finding F-1-11

c,
&format!(
"MATCH (p:{label}) \
SEARCH p IN (FULLTEXT INDEX idx{label} FOR '{query}' LIMIT 50) \

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] comparison (flag-canonical-dup): The speaker corpus re-copies the people-store drivers: speaker_search is search_names with only the label, index and LIMIT (Person/personText/10) substituted, and speaker_procedure is procedure_names

Suggested fix: Parameterize the corpus (label, index, limit) in one SEARCH-clause driver and one queryNodes driver, make the people and speaker call sites thin wrappers, and keep speakers() as the one dual-entry-point assertion.

finding F-1-12

pub(super) fn parse(term: &str) -> Option<Self> {
let mut parts = Vec::new();
let mut wild = false;
let mut chars = term.chars().flat_map(char::to_lowercase);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] comparison (std-6): Pattern::parse is matched against stored tokens, which tokenize lowercases with str::to_lowercase (fulltext.rs:246), but parse lowercases per char with char::to_lowercase. The two primitives differ wh

Suggested fix: Lowercase the term once with the stored side's primitive (let lowered = term.to_lowercase(); then iterate lowered.chars()) so pattern atoms and the tokens they match come from one lowercasing rule; leave the wildcard's no-stemming/no-folding policy unchanged.

finding F-1-13

if quoted && matches!(c, '*' | '?') {
current.push('\\');
}
current.push(c);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] branching (std-2): Quotedness is carried out of the tokenizer as an injected backslash character that Pattern::parse later un-escapes, so the quoting policy is encoded in the word text instead of in the word itself: a

Suggested fix: Yield the word with its quoting as data (a small Word { text, quoted }), and let push_text skip only the wildcard branch for a quoted word; delete the escape injection and its branch.

finding F-1-5

Error::execution(format!(
"InvalidIndexDefinition: fulltext index {index_name} has an invalid indexConfig option"
))
};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] boundaries (std-6): The new fulltext read path spells its own drift rejection, while exec::vector::invalid_index_option is documented as "One drift contract for every invalid stored index option" and renders the identi

Suggested fix: Promote the drift constructor to the shared scan module (exec::candidates, already the home of resolve_search_index), parameterized by kind and index name, and call it from both exec::vector::stored_config and index_analyzer; the local closure then disappears.

finding F-1-6

crate::exec::ddl_options::index_analyzer(&options, &index_name)?,
None,
);
let produced = fulltext::search_all(graph, &detail, &query, analyzers)?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] boundaries (std-6): The fulltext catalog phase — resolve_search_index (lines 650-651), the stored-analyzer read, Analyzers::new, EntityKind::from_relationship_flag, and candidate materialization — is orchestrated a

Suggested fix: Extract the shared kernel into exec::fulltext (mirroring vector_index_candidates): resolve the index, build the Analyzers from the stored options plus an optional override, and return (EntityKind, Vec<(Value, f64)>); both fulltext_index_candidates and exec_fulltext_query_call call it.

finding F-1-7

"MATCH (p:{label}) \
SEARCH p IN (FULLTEXT INDEX idx{label} FOR '{query}' LIMIT 50) \
SCORE AS score RETURN p.name AS name, score ORDER BY score DESC, name ASC"
),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] boundaries (std-6): speaker_search (line 534) re-implements the pre-existing search_names SEARCH template, differing only in the label/index and LIMIT; a second copy of one query shape is already drifting (LIMIT 10 v

Suggested fix: Parameterize the existing helper — search_names(c, label, index, limit, query) — and call it from both the speaker/folding corpus and the fuzzy/boost tests instead of adding a second builder.

finding F-1-8

c,
&format!(
"CALL db.index.fulltext.queryNodes('idx{label}', '{query}') \
YIELD node, score RETURN node.name AS name, score ORDER BY score DESC, name ASC"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] boundaries (std-6): speaker_procedure is the scored form of the pre-existing procedure_names over the same queryNodes template; only the label/index and the score projection differ, so the file now spells one CALL

Suggested fix: Fold both into one scored helper parameterized by label/index, and project the names where the callers need them.

finding F-1-9

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Stellar review (stellar-review skill)

Round 8 · head 30055863 · scope: 0/203 units (203 sealed from earlier rounds).

Still open from earlier rounds

  • [Medium] structure (std-6) F-1-1: relate/relate-executor/src/exec/ddl_options.rs:428-431 — This is the fulltext scan's stored-config read — it turns a stored indexConfig value into a query-time Analyzer and raises a query-time drift error — but it lives in the schema-DDL options module,
  • [Medium] comparison (flag-canonical-dup) F-1-10: relate/relate-executor/src/exec/ddl_options.rs:432-436 — The stored-option drift contract already exists as exec::vector::invalid_index_option (vector.rs:172-176), whose doc claims it is "One drift contract for every invalid stored index option" and whose s
  • [Medium] comparison (std-5) F-1-11: relate/relate-executor/src/exec/search.rs:119-124 — WITH ANALYZER's two spellings of "no analyzer named" now diverge: an absent clause reaches Analyzers::new as None and follows the index (Analyzer::query_default), while an explicit NULL is lowered to
  • [Medium] comparison (flag-canonical-dup) F-1-12: relate/relate/tests/fulltext_search_semantics.rs:534-539 — The speaker corpus re-copies the people-store drivers: speaker_search is search_names with only the label, index and LIMIT (Person/personText/10) substituted, and speaker_procedure is procedure_names
  • [Medium] comparison (std-6) F-1-13: relate/relate-executor/src/exec/fulltext/wildcard.rs:22-25 — Pattern::parse is matched against stored tokens, which tokenize lowercases with str::to_lowercase (fulltext.rs:246), but parse lowercases per char with char::to_lowercase. The two primitives differ wh
  • [Low] structure (q-legibility) F-1-14: relate/relate-executor/src/exec/fulltext.rs:190-194 — The PR appends stored_tokens and query_default in a second impl Analyzer block that opens immediately after the first one closes, splitting one type's methods across two blocks with nothing betw
  • [Low] structure (q-legibility) F-1-15: relate/relate-executor/src/exec/fulltext.rs:2-3 — The module overview still names standard and english as the two differentiated search analyzers; this PR gives standard-folding its own behaviour, so the header both omits the new differentiated
  • [Low] branching (std-3) F-1-16: relate/relate-executor/src/exec/fulltext.rs:469-474 — The let alternatives = match field_match reads as a value extraction, but the Constant arm is actually scoring work ended by a continue, so the two variants' scoring lives in two different shape
  • [Low] branching (std-3) F-1-17: relate/relate-executor/src/exec/fulltext/wildcard.rs:27-32 — The **-is-* rule is implemented as a continue from inside a parts.push(match ...) argument, so this arm never yields a value and control leaves from the middle of an expression — the one place
  • [Low] branching (std-3) F-1-18: relate/relate/tests/fulltext_search_semantics.rs:712-714 — The test's subject is backslash and quote handling, yet every query crosses two escape layers (Rust literal -> Cypher string literal), so r"kollegger\\*" is the query kollegger\* and `r#""kolleg*
  • [Low] boundaries (std-5) F-1-19: relate/relate-executor/src/exec/search.rs:115-118 — The comment documents the optional slot's default as "the standard analyzer", but the code passes None to Analyzers::new, which selects Analyzer::query_default() — the index's folding for a `sta
  • [Medium] structure (std-1) F-1-2: relate/relate-executor/src/exec/fulltext.rs:50-54 — fulltext.rs lands at 813 lines after +230 in this PR, and it now carries two distinct vocabularies: the analyzer layer (Analyzer, Analyzers, the 45-row ANALYZERS table, tokenize, `analyzer_nam
  • [Low] boundaries (std-5) F-1-20: relate/relate-executor/src/exec/fulltext.rs:204-209 — query_default is pub(crate) but its only caller is Analyzers::new in the same module (stored_tokens beside it is correctly private), so this widens the analyzer's surface for a consumer that d
  • [Low] comparison (q-visibility) F-1-21: relate/relate-executor/src/exec/fulltext.rs:204-204 — The diff's new items all take least visibility: fold, Pattern and Target are pub(super) for the parent module, and each pub(crate) item (Analyzers, Analyzers::new, index_analyzer, KEY_FULLTEXT_ANALYZE
  • [Medium] branching (std-2) F-1-3: relate/relate-executor/src/exec/fulltext.rs:194-198 — stored_tokens and query_default are the same projection of the same three variants — "the stored side drops the query-side stop words and stems but keeps folding" — spelled as two different matche
  • [Medium] branching (flag-bool-mode) F-1-4: relate/relate-executor/src/exec/procedures.rs:626-628 — The entity kind arrives as a raw bool and this executor re-derives three separate facts from it: the yielded column spelling here, detail.relationship != relationship at line 652, and `EntityKind::f
  • [Medium] branching (std-2) F-1-5: relate/relate-executor/src/exec/fulltext/query.rs:123-128 — Quotedness is carried out of the tokenizer as an injected backslash character that Pattern::parse later un-escapes, so the quoting policy is encoded in the word text instead of in the word itself: a
  • [Medium] boundaries (std-6) F-1-6: relate/relate-executor/src/exec/ddl_options.rs:432-436 — The new fulltext read path spells its own drift rejection, while exec::vector::invalid_index_option is documented as "One drift contract for every invalid stored index option" and renders the identi
  • [Medium] boundaries (std-6) F-1-7: relate/relate-executor/src/exec/procedures.rs:660-665 — The fulltext catalog phase — resolve_search_index (lines 650-651), the stored-analyzer read, Analyzers::new, EntityKind::from_relationship_flag, and candidate materialization — is orchestrated a
  • [Medium] boundaries (std-6) F-1-8: relate/relate/tests/fulltext_search_semantics.rs:538-541 — speaker_search (line 534) re-implements the pre-existing search_names SEARCH template, differing only in the label/index and LIMIT; a second copy of one query shape is already drifting (LIMIT 10 v
  • [Medium] boundaries (std-6) F-1-9: relate/relate/tests/fulltext_search_semantics.rs:546-551 — speaker_procedure is the scored form of the pre-existing procedure_names over the same queryNodes template; only the label/index and the score projection differ, so the file now spells one CALL
  • [Medium] structure (std-6) F-2-1: relate/relate-executor/src/exec/fulltext.rs:194-196 — The analyzer exposes its folding policy as a bare flag whose only consumer is the query parser (query.rs:207), so the parser reconstructs the analyzer's term preparation itself — `fold(&term.to_lowerc
  • [Medium] branching (std-0) F-2-2: relate/relate-executor/src/exec/fulltext.rs:194-196 — folds is the fourth independent projection of Analyzer's three variants — analyze, stored_tokens and query_default each re-decide folding/stop-words/stemming in their own match — and it ex
  • [Low] boundaries (q-visibility) F-2-3: relate/relate-executor/src/exec/fulltext.rs:194-196 — The method's only caller is fulltext::query (query.rs:207), a child module that already reaches the private analyze the same way; pub(crate) needlessly widens Analyzer's surface to every modul
  • [Medium] branching (std-2) F-5-1: relate/relate-executor/src/exec/fulltext/query.rs:168-171 — The suffix grammar has no single owner: this new glue stage decides "detached suffix" by spelling the marker set as starts_with(['~', '^']), while split_suffix (line 287) independently re-lists th
  • [Medium] boundaries (std-5) F-5-2: relate/relate-executor/src/exec/fulltext/query.rs:157-159 — quoted is scanned state that is never inspected at end of input: an unterminated quote is silently flushed as if it had closed, while this same Result-returning lexer raises a typed malformed er
  • [Medium] boundaries (std-5) F-5-3: relate/relate-executor/src/exec/fulltext/query.rs:487-489 — Both assertions (here and the lamp\~1 check below) are satisfiable by the broken parse the test exists to catch: the escaped \:/\~ reaches tokenize, which splits on the backslash, so a\:b yi
  • [Medium] boundaries (std-5) F-5-4: relate/relate-executor/src/exec/fulltext/query.rs:212-215 — The changed split now finds the delimiter past escaped colons, but the property half keeps those escapes: a\:b:term yields a clause property of a\:b, which resolve_clause compares raw against fi
  • [Low] boundaries (std-5) F-5-5: relate/relate-executor/src/exec/fulltext/query.rs:479-480 — The test name promises a typed syntax error but the assertion only matches rendered message text, so lex could return any Error variant (or the wording drift) and this test would behave identicall

Stellar review: ⚠️ 26 issue(s) open (0 + 0 anchored new (fail/low) + 26 carried)

@jexp
jexp force-pushed the feat/fulltext-fold-wildcard branch from 3104db1 to f917343 Compare November 12, 2026 03:02

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stellar review round 2: 2 inline finding(s); see summary comment.

/// and, by ruling, wildcard terms).
pub(crate) fn folds(self) -> bool {
matches!(self, Self::StandardFolding)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] structure (std-6): The analyzer exposes its folding policy as a bare flag whose only consumer is the query parser (query.rs:207), so the parser reconstructs the analyzer's term preparation itself — `fold(&term.to_lowerc

Suggested fix: Give Analyzer one preparation entry point (e.g. fn term_text(self, term: &str) -> String that lowercases and folds when the variant folds) and have query.rs call it unconditionally, deleting the folds() branch and the parser's fold import; the flag then disappears from the analyzer's surfa

finding F-2-1

/// and, by ruling, wildcard terms).
pub(crate) fn folds(self) -> bool {
matches!(self, Self::StandardFolding)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] branching (std-0): folds is the fourth independent projection of Analyzer's three variants — analyze, stored_tokens and query_default each re-decide folding/stop-words/stemming in their own match — and it ex

Suggested fix: Make the behaviour data instead of four matches: Analyzer carries the flags its 45-row ANALYZERS table already implies (fold / stop_words / stems) behind one token pipeline, so folds becomes a field read, analyze/stored_tokens/query_default stop re-deciding the same policy, and the

finding F-2-2

))
};
let config = index_config(options).map_err(|_| drift())?;
match config.and_then(|config| config.get(KEY_FULLTEXT_ANALYZER)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] Error handling: index_analyzer reads fulltext.analyzer with an exact-case lookup while create-time validation admits the key in any case, so an accepted FULLTEXT.ANALYZER config is silently ignored and the index behaves as standard.

Why this matters: validate_config_family lowercases indexConfig keys for admission (ddl_options.rs:534), but apply_fulltext_defaults then inserts the canonical lowercase fulltext.analyzer entry, so this read finds the default instead of the user's standard-folding and stored tokens are never folded — a silent wrong-result path for a DDL statement the engine accepted.
Suggested fix: normalize the keys once at create, or look this key up case-insensitively (for example config.iter().find(|(key, _)| key.eq_ignore_ascii_case(KEY_FULLTEXT_ANALYZER)).map(|(_, value)| value)).

@jexp
jexp marked this pull request as ready for review November 12, 2026 06:59
@@ -117,14 +116,19 @@ fn fulltext_index_candidates(
// FulltextSearchAcceptance [57]); an absent slot means the standard

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low — Docs: The comment still says an absent WITH ANALYZER slot means the standard analyzer.

Why this matters: Analyzers::new(index, None) now resolves to Analyzer::query_default(), so a standard-folding index folds query terms even without WITH ANALYZER; the exec/fulltext.rs module header carries the same drift, still describing "the two differentiated search analyzers (standard, english)" though standard-folding is now a third.
Suggested fix: Say "absent means the index's query default (its folding, not its stemming)" here, and update the module header to name three analyzers.

))
};
let config = index_config(options).map_err(|_| drift())?;
match config.and_then(|config| config.get(KEY_FULLTEXT_ANALYZER)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low — Silent fallback: The stored indexConfig read matches fulltext.analyzer by exact case, but validate_config_family admits the key in any casing.

Why this matters: an index created with the key spelled FULLTEXT.ANALYZER passes validation, then apply_fulltext_defaults adds the canonical fulltext.analyzer default next to it, so this lookup silently returns the default and the configured analyzer (for example standard-folding) is never applied to queries.
Suggested fix: Look the key up case-insensitively (for example config.iter().find(|(key, _)| key.eq_ignore_ascii_case(KEY_FULLTEXT_ANALYZER))), or normalise the stored config keys at the create boundary.

@jexp
jexp force-pushed the feat/fulltext-fuzzy-boost branch from b36a6a3 to 212ebb1 Compare November 12, 2026 10:57
@jexp
jexp force-pushed the feat/fulltext-fold-wildcard branch from f917343 to 81422a0 Compare November 12, 2026 11:01
Target::Fuzzy { term, edits } => {
FieldMatch::Weighted(expand_fuzzy(term, *edits, stat))
}
Target::Wildcard(pattern) => FieldMatch::Constant(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] Denial of service (resource exhaustion): each wildcard clause clones its whole matching field vocabulary into a FieldMatch::Constant, so query memory grows as (wildcard clauses × distinct indexed terms).

Why this matters: the fulltext query text is caller-controlled and the 4 MiB statement cap is no bound here — * * * … costs two bytes per clause, and every clause independently retains a full vocabulary copy, so a ~10 KB read-only query (SEARCH … FULLTEXT INDEX or db.index.fulltext.queryNodes) against a 100k-term index retains hundreds of millions of cloned strings and OOMs the process. Pattern::matches also costs O(pattern × term) per vocabulary term, so one long ? pattern adds a CPU multiplier on top.
Suggested fix: expand each distinct pattern once per scan and share the set across clauses, and bound total expanded terms (or wildcard clause count) per query with a typed error rather than silent truncation, keeping ADR-0583's no-silent-drop constraint.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stellar review round 4: 5 inline finding(s); see summary comment.

}
}
words.push(word.to_string());
words.push(word);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] structure (std-6): This round's rewrite moved the detached-suffix rewrite into the flush step, so emitting a word also reaches back and mutates the previously emitted one — and to decide whether it may, the scanner ha

Suggested fix: Make words one stage: scan to whole words (escapes and quotes only) and return them, then attach a detached ~/^ word to its predecessor in a separate pass — e.g. fn attach_suffixes(words: &mut Vec<String>) called from parse next to Parser::word's operator folding — so the emission step o

finding F-4-1

} else {
current.push(other);
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] branching (q-missing-helper): The escape arm re-decides the word-boundary policy the main loop owns eight lines below (} else if c.is_whitespace() || matches!(c, '(' | ')') {): the separator test now exists in two places, and an

Suggested fix: Peel the escape once and hand every character — escaped or not — to the same disposition (flush on a separator, else append), so the separator test exists in one place; if the scan keeps its shape, extract fn is_separator(c: char) -> bool and call it at both sites.

finding F-4-2

Some(next @ ('*' | '?' | '\\')) => {
current.push('\\');
current.push(next);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] branching (std-0): The escape match spells "this character is a literal in pattern text" by hand in each arm — an escaped metacharacter as a backslash push plus the char, an escaped backslash as a literal two-backslash

Suggested fix: Put the one way to append a literal character to pattern text in wildcard (e.g. Pattern::push_literal(&mut String, char), escaping the metacharacters) and route the escape arms through it: the metachar and escaped-quote arms collapse into one push_literal(next) arm, and the trailing-escape and

finding F-4-3

} else {
current.push(other);
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] boundaries (std-5): The arm above promises any escaped character "stands for itself", but it consumes the backslash and hand the bare character on: Parser::word still splits an escaped a\:b at the colon, flush/`spl

Suggested fix: Carry the word as typed text (chars with a literal flag, or a Word value) and decode escapes once at the boundary so Parser::word and split_suffix consult literalness instead of re-splitting a string; alternatively escape every downstream metacharacter at this one site. Extend the new escape t

finding F-4-4

current.push(other);
}
}
None => current.push_str("\\\\"),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] boundaries (std-5): An escape with no following character is silently repaired: instead of the typed malformed syntax error every other malformed suffix raises, the lexer fabricates a literal backslash, which `tokenize

Suggested fix: Return Result<Vec<String>> from words (its only caller parse is already fallible) and raise malformed(query, ...) for a dangling escape; change the test's "does not panic" case to assert the typed syntax error.

finding F-4-5

pub(super) fn parse(term: &str) -> Option<Self> {
let mut parts = Vec::new();
let mut wild = false;
let mut chars = term.chars().flat_map(char::to_lowercase);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] Correctness: Pattern::parse lowercases with char::to_lowercase while stored tokens are lowercased by str::to_lowercase in tokenize; the two disagree on Greek final sigma.

Why this matters: "ΑΣ" is stored as ας, but the pattern ΑΣ* lowercases to ασ*, so on a non-folding index that wildcard never matches its own stored token while the plain term ΑΣ does.
Suggested fix: lowercase the term as a string first (let lowered = term.to_lowercase(); then iterate lowered.chars()), so the wildcard path shares the tokenizer's lowercase policy.

fulltext::Analyzer::from_value(&eval(graph, input, expr, params, cypher_version)?)
})
.transpose()?;
let analyzers = fulltext::Analyzers::new(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] Option handling: an explicit WITH ANALYZER NULL evaluates to Analyzer::Standard and is passed as Some(..), so it bypasses query_default() and turns off the index's folding.

Why this matters: Analyzer::from_value documents NULL as CIP-225's "no override", yet on a standard-folding index FOR 'eifrem' WITH ANALYZER NULL now misses Eifrém while omitting WITH ANALYZER finds it — the two forms silently diverge, and the comment just above (line 116) still says "an absent slot means the standard analyzer".
Suggested fix: collapse an evaluated Value::Null to None before Analyzers::new (e.g. match the evaluated value: Value::Null => None, other => Some(Analyzer::from_value(&other)?)), and add a folding-index test for WITH ANALYZER NULL.

@jexp
jexp force-pushed the feat/fulltext-fold-wildcard branch from 81422a0 to d8d0ca9 Compare November 12, 2026 12:53
Target::Fuzzy { term, edits } => {
FieldMatch::Weighted(expand_fuzzy(term, *edits, stat))
}
Target::Wildcard(pattern) => FieldMatch::Constant(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] Denial of service: Every wildcard clause independently clones all matching vocabulary terms into its own HashSet, uncapped and uncharged by the ADR-0463 allocation budget.

Why this matters: the SEARCH query (and the fulltext procedures) is client-controlled text on every surface, so a ~2 KB parameter of 1,000 * clauses retains 1,000 × |vocabulary| owned strings — gigabytes on a 100k-term index, before scoring starts — letting a tiny query OOM the process past the configured cap.

Suggested fix: resolve each distinct (pattern, field) expansion once per scan and charge the retained set via ExecGraph::check_expression_allocation; sharing does not truncate hits, so the ADR's no-cap parity goal still holds.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stellar review round 5: 4 inline finding(s); see summary comment.

last.push_str(word);
if let Some(last) = out.last_mut().filter(|last| operator(last).is_none()) {
last.push_str(&word);
continue;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] branching (std-2): The suffix grammar has no single owner: this new glue stage decides "detached suffix" by spelling the marker set as starts_with(['~', '^']), while split_suffix (line 287) independently re-lists th

Suggested fix: Put the marker vocabulary beside the Suffix type and have both stages ask it — e.g. impl Suffix { fn begins(word: &str) -> bool; fn find(text: &str) -> Option<usize> }, or one const SUFFIX_MARKERS: [char; 2] consulted by attach_suffixes and by split_suffix's find_unescaped call — so the

finding F-5-1

}
}
flush(&mut current);
Ok(words)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] boundaries (std-5): quoted is scanned state that is never inspected at end of input: an unterminated quote is silently flushed as if it had closed, while this same Result-returning lexer raises a typed malformed er

Suggested fix: Decide the invariant and apply it once at EOF: raise malformed(query, "an unterminated quote") when quoted is still true after the loop (live's Lucene lexer rejects it), or, if accepting it is deliberate, state that shortcut in the module-header list and pin it with a test; either way the rule l

finding F-5-2

// quoted `f:x` has no property either.
for query in [r"a\:b", r#""f:x""#] {
let parsed = clauses(query);
assert!(parsed.iter().all(|c| c.property.is_none()), "{query}");

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] boundaries (std-5): Both assertions (here and the lamp\~1 check below) are satisfiable by the broken parse the test exists to catch: the escaped \:/\~ reaches tokenize, which splits on the backslash, so a\:b yi

Suggested fix: Assert the positive contract instead of the absence of two wrong shapes: assert_eq!(parsed.len(), 1) plus assert_eq!(term(&parsed[0]), "a:b"), and the equivalent single-term/Fuzzy assertions for lamp\~1, so the test fails until escaped characters are actually carried inside one word.

finding F-5-3

let (property, text) = match find_unescaped(word, &[':']) {
Some(at) => (Some(&word[..at]), &word[at + 1..]),
None => (None, word),
};

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] boundaries (std-5): The changed split now finds the delimiter past escaped colons, but the property half keeps those escapes: a\:b:term yields a clause property of a\:b, which resolve_clause compares raw against fi

Suggested fix: Give both halves of the split the same decoding: run the property slice through the same escape pass the term text uses (or resolve it to a typed property name that owns the decoding) so a\:b:term names the field a:b, rather than emitting a property string no field can ever equal.

finding F-5-4

@jexp
jexp force-pushed the feat/fulltext-fuzzy-boost branch from 212ebb1 to f258817 Compare November 12, 2026 13:44
@jexp

jexp commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Deferred review findings → LABS-1491 (review-tasks, Agent Backlog) with a Files: scope, per maintainer direction 2026-11-12: merge now, fix the findings as follow-up work. No blocker/critical findings are among them.

Base automatically changed from feat/fulltext-fuzzy-boost to main November 12, 2026 14:08
@jexp
jexp force-pushed the feat/fulltext-fold-wildcard branch from d8d0ca9 to bf341cc Compare November 12, 2026 14:09
for c in token.chars() {
if c.is_ascii() {
out.push(c);
} else if unicode_normalization::char::is_combining_mark(c) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] Correctness: dropping standalone combining marks makes the fold depend on the Unicode normal form of non-Latin input.

Why this matters: this branch drops every standalone combining mark, so NFD Greek α + tonos folds to α while the NFC form ά passes through unchanged. Two canonically equivalent spellings of the same word then produce different stored tokens, contradicting the documented "Greek and Cyrillic are untouched" guarantee (the NFKD branch comment below makes the same claim).

Suggested fix: drop the mark only when the base it follows was actually folded to ASCII (track whether the previous output came from an ASCII fold); otherwise keep the original character with its marks.

pub(super) fn parse(term: &str) -> Option<Self> {
let mut parts = Vec::new();
let mut wild = false;
let mut chars = term.chars().flat_map(char::to_lowercase);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] Idiomatic Rust: wildcard terms are lowercased per char while stored tokens are lowercased with str::to_lowercase, so the two disagree on the Greek final sigma.

Why this matters: str::to_lowercase applies the context-sensitive final-sigma rule (a capital Σ at a word end becomes ς), but char::to_lowercase always yields σ. The stored token for ΣΟΦΟΣ is σοφος, while Pattern::parse("ΣΟΦΟΣ*") builds σοφοσ*, so the prefix wildcard never matches the word that the exact term ΣΟΦΟΣ matches.

Suggested fix: lowercase the whole term first (let lowered = term.to_lowercase();) and iterate lowered.chars(); escapes, * and ? are unaffected by case mapping.

.analyzer
.as_ref()
.map(|expr| {
fulltext::Analyzer::from_value(&eval(graph, input, expr, params, cypher_version)?)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] Option handling: WITH ANALYZER NULL is silently turned into an explicit Standard override, so it stops following the index folding.

Why this matters: Analyzer::from_value documents NULL as "no override" and returns Standard; wrapping that in Some(...) makes Analyzers::new treat it as an override. On a standard-folding index, FOR 'eifrem' WITH ANALYZER NULL no longer folds and misses Eifrém, while the same query without the clause finds it. The comment above ("an absent slot means the standard analyzer") is also stale, since an absent slot now means the index default.

Suggested fix: make the override Option<Analyzer> end to end (NULL -> None, so Analyzers::new falls back to index.query_default()), or map NULL to the index analyzer here, and update the comments.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Spec-harness coverage — 🟡

  • 🟡 spec coverage 99.41% — 200 genuine failures (0.59% of the executed scenarios)
  • scenarios run: 33639 (passed 33439, failed 200)
  • bands: 🟢 zero failures · 🟡 failure share ≤ 10% · 🔴 beyond (ADR-0482 §4 — the light never fails the build)
  • suite runtime: 53s
  • neo4j_expected_fail (not counted): 412
  • nondeterministic (not counted): 5
  • skipped (not counted): 2323
  • harness ref: 5a89fd3425c37acd057ff67dab016c5c73aff240
failure category count
result-mismatch 32
syntax-error 31
error-class-mismatch 30
result-temporal-fidelity 29
missing-validation 27
notification-missing 19
semantic-error 10
type-error 10
other 9
setup-failed-syntax 3

Rerun spec-harness to refresh. Coverage below 100% never fails the build (ADR-0482 §4).

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Loadtest baselines comparison

Warning: loadtest comparison reported problems — see the flagged pairs below.


Loadtest baseline comparison

Frozen baselines vs current reports; bands match the compare-loadtests gate.

Status: warning

Pairs: fraud-loadtest: compared · identity-loadtest: compared (failing rows)

fraud-loadtest

baselines/fraud-loadtest.json → target/fraud-loadtest.json

Metric Baseline PR Δ Status
environment relate-loadtest-baseline/v1 relate-loadtest-baseline/v1 — match
errors 0 0 +0 ok
ops/s 3938.56 3965.89 +27.33 (+0.7%) ok
p99 (worst: p2p_transfer) 8.04 ms 7.80 ms -0.24 ms (-3.0%) ok
logical bytes 17,344,481 16,400,573 -943,908 (-5.4%) ok
Per-use-case p99 latency
Use case ops Baseline p99 PR p99 Δ Status
p2p_transfer 3,859 8.04 ms 7.80 ms -0.24 ms (-3.0%) ok
payroll_scatter 1,139 17.85 ms 16.33 ms -1.52 ms (-8.5%) ok
onboard_customer 806 8.53 ms 7.39 ms -1.14 ms (-13.4%) ok
mule_scatter 814 14.80 ms 11.20 ms -3.60 ms (-24.3%) ok
layering_cycle 822 15.10 ms 10.77 ms -4.33 ms (-28.7%) ok
recent_transfers 2,199 7.65 ms 7.34 ms -0.32 ms (-4.1%) ok
customer_portfolio 1,376 7.37 ms 6.56 ms -0.81 ms (-11.0%) ok
outgoing_volume 1,093 7.48 ms 6.83 ms -0.65 ms (-8.6%) ok
detect_fanout 830 25.78 ms 12.60 ms -13.17 ms (-51.1%) ok
detect_cycle 843 7.55 ms 6.71 ms -0.84 ms (-11.1%) ok
activity_page 1,123 7.48 ms 6.70 ms -0.78 ms (-10.4%) ok
search_customer 927 7.84 ms 7.09 ms -0.75 ms (-9.6%) ok
counterparties 1,059 7.45 ms 6.61 ms -0.84 ms (-11.2%) ok
payment_path 849 7.47 ms 6.97 ms -0.50 ms (-6.8%) ok
amount_band 860 7.30 ms 6.81 ms -0.49 ms (-6.7%) ok
flag_account 732 8.21 ms 7.21 ms -0.99 ms (-12.1%) ok
reverse_last_out 669 8.04 ms 7.67 ms -0.37 ms (-4.6%) ok

identity-loadtest

baselines/identity-loadtest.json → target/identity-loadtest.json

Metric Baseline PR Δ Status
environment relate-loadtest-baseline/v1 relate-loadtest-baseline/v1 — match
errors 0 2 +2 fail (2 errors)
ops/s 573.51 731.45 +157.94 (+27.5%) ok
p99 (worst: group_roster) 7.72 ms 7.78 ms +0.06 ms (+0.8%) ok
logical bytes 7,167,614 7,392,964 +225,350 (+3.1%) ok
Per-use-case p99 latency
Use case ops Baseline p99 PR p99 Δ Status
provision_user 1,373 11.00 ms 9.14 ms -1.85 ms (-16.8%) ok
add_to_group 1,775 18.57 ms 15.27 ms -3.29 ms (-17.7%) ok
grant_role 1,079 18.52 ms 15.77 ms -2.74 ms (-14.8%) ok
attach_device 907 18.88 ms 15.60 ms -3.28 ms (-17.4%) ok
nested_grant 736 19.41 ms 15.62 ms -3.79 ms (-19.5%) ok
effective_access 2,173 25.25 ms 17.82 ms -7.43 ms (-29.4%) ok
org_walk 1,387 19.94 ms 15.66 ms -4.28 ms (-21.5%) ok
optional_manager 1,312 20.00 ms 16.00 ms -4.00 ms (-20.0%) ok
detect_nested_admin 810 19.97 ms 15.33 ms -4.64 ms (-23.2%) ok
detect_sod 817 19.60 ms 14.78 ms -4.82 ms (-24.6%) ok
stale_grants 827 15.06 ms 12.57 ms -2.49 ms (-16.5%) ok
search_directory 1,147 10.91 ms 9.57 ms -1.34 ms (-12.3%) ok
group_roster 1,047 7.72 ms 7.78 ms +0.06 ms (+0.8%) ok
access_review 801 20.02 ms 16.13 ms -3.90 ms (-19.5%) ok
reporting_path 794 30.98 ms 20.75 ms -10.23 ms (-33.0%) ok
disable_user 644 19.50 ms 15.06 ms -4.44 ms (-22.8%) ok
add_alias 710 19.99 ms 15.72 ms -4.27 ms (-21.4%) ok
revoke_membership 674 8.10 ms 7.28 ms -0.82 ms (-10.1%) ok
detach_device 538 14.79 ms 10.13 ms -4.67 ms (-31.5%) ok
clear_expiry 449 17.39 ms 14.11 ms -3.28 ms (-18.9%) ok

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Memory controls and workspace memory calibration

Warning: the comparison document target/memory-controls.md is missing — the render step produced nothing this run.


(comparison document unavailable)

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Memory shape

Note: loadtest comparison completed with no failing rows.


Memory shape

Status: ok

All memory shapes held. 🟢

base

Given: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling.

When: the base shape is loaded to 0 nodes and 0 relationships, then checkpointed. The child RSS is sampled.

Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries.

Baseline: logical 0.00 MB – 0.00 MB, disk 0.00 MB – 0.00 MB, attributed RSS 0.00 MB – 0.00 MB, ratio 0.000x–0.000x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes.

Validated: logical 0.00 MB, disk 0.00 MB, attributed RSS 0.00 MB, ratio 0.000x.

Report: logical, disk, attributed RSS, and the ratio held.

flat

Given: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling.

When: the flat shape is loaded to 120000 nodes and 0 relationships, then checkpointed. The child RSS is sampled.

Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries.

Baseline: logical 26.02 MB – 26.02 MB, disk 1.95 MB – 1.95 MB, attributed RSS 36.49 MB – 36.93 MB, ratio 1.402x–1.419x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes.

Validated: logical 26.02 MB, disk 1.26 MB, attributed RSS 37.73 MB, ratio 1.450x.

Report: logical, disk, attributed RSS, and the ratio held.

fanout

Given: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling.

When: the fanout shape is loaded to 52016 nodes and 52000 relationships, then checkpointed. The child RSS is sampled.

Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries.

Baseline: logical 28.62 MB – 28.62 MB, disk 1.75 MB – 1.75 MB, attributed RSS 34.93 MB – 36.61 MB, ratio 1.220x–1.279x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes.

Validated: logical 27.43 MB, disk 1.10 MB, attributed RSS 35.94 MB, ratio 1.310x.

Report: logical, disk, attributed RSS, and the ratio held.

ring

Given: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling.

When: the ring shape is loaded to 46000 nodes and 46000 relationships, then checkpointed. The child RSS is sampled.

Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries.

Baseline: logical 29.99 MB – 29.99 MB, disk 1.77 MB – 1.77 MB, attributed RSS 35.30 MB – 35.89 MB, ratio 1.177x–1.197x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes.

Validated: logical 29.99 MB, disk 1.20 MB, attributed RSS 35.68 MB, ratio 1.190x.

Report: logical, disk, attributed RSS, and the ratio held.

dense ring

Given: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling.

When: the dense ring shape is loaded to 5000 nodes and 100000 relationships, then checkpointed. The child RSS is sampled.

Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries.

Baseline: logical 31.41 MB – 31.41 MB, disk 1.62 MB – 1.62 MB, attributed RSS 31.27 MB – 32.46 MB, ratio 0.996x–1.034x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes.

Validated: logical 31.11 MB, disk 0.73 MB, attributed RSS 31.87 MB, ratio 1.024x.

Report: logical, disk, attributed RSS, and the ratio held.

@jexp
jexp force-pushed the feat/fulltext-fold-wildcard branch from bf341cc to 20f132c Compare November 12, 2026 14:42
@jexp

jexp commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (20f132cd) — with one caveat worth recording: the previous head silently reverted main's #244 and #245 fixes (the creation_failure wrapper in constraint_validation/matrix.rs and the ddl test's EquivalentSchemaRuleAlreadyExists token), because its merge-base predated them and the merge was textually clean. This commit touches only its own 10 files, so it was cherry-picked onto current main; set-equality is now exactly those 10 with zero reverts, suites green (executor + relate), fmt and clippy clean, stellar ledger fully sealed from earlier rounds (findings deferred → LABS-1491).

@oplane-bot

oplane-bot Bot commented Nov 12, 2026

Copy link
Copy Markdown

Oplane Security Review

Oplane identified security requirements relevant to this PR and checked the implementation against them, finding that 1 requirement needs attention. Push changes to rerun the review, update a status with a justification if needed, or review the generated threat model.

StatusRequirementSeverityAction
🔴 Not implementedInput Validation and Wildcard Pattern Restriction for Fulltext Query Strings
OPLANE_REQ-00357763
LowPrompt · Claude · Cursor
🟢 ImplementedInput Sanitization and Escaping for Wildcard and Quoted Query Terms
OPLANE_REQ-00357764
High

Change comment settings · Set up Oplane in your agent · Workflow guidance

A standard-folding index now folds Latin diacritics and ligatures in its
stored tokens and in analyzed query terms (the index's configured
analyzer is read). Wildcard terms (term*, te?m, *term, field:term*,
*:*) expand over each field's vocabulary into the any-of group the fuzzy
path uses and score a constant 1.0 per matching field times the boost,
with no expansion cap. Under a folding analyzer the literal parts of a
wildcard are folded like stored tokens (a deliberate deviation from live
Neo4j, which leaves them unfolded); fuzzy terms stay lowercased only, and
escaped or quoted wildcard characters are plain text. Calibrated against
Neo4j 2026.09-enterprise.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@jexp
jexp force-pushed the feat/fulltext-fold-wildcard branch from 20f132c to 3005586 Compare November 12, 2026 14:44
Target::Fuzzy { term, edits } => {
FieldMatch::Weighted(expand_fuzzy(term, *edits, stat))
}
Target::Wildcard(pattern) => FieldMatch::Constant(

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Medium] Denial of service: Uncapped wildcard expansion is rebuilt per clause, so query-time memory grows as clauses × field vocabulary.

Why this matters: each wildcard clause clones every matching term into its own HashSet<String> for every indexed field; clause count is user-controlled (a 4 MiB statement or a large db.index.fulltext.queryNodes parameter holds ~2M * clauses), so a long query over a small index exhausts RSS and OOMs the process, bypassing the existing budget seam (ADR-0463) and fuzzy's 50-term cap.

Suggested fix: bound the expansion per field as MAX_FUZZY_EXPANSIONS does, or share one expansion per (pattern, field) and charge it to the statement budget.

@jexp

jexp commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Deferred security-review finding → LABS-1498 (review-tasks, Agent Backlog, Files: scope), per maintainer direction 2026-11-12: [Low] wildcard pattern restriction policy. The High requirement (input sanitization and escaping for wildcard/quoted terms) is implemented. Stellar findings already deferred → LABS-1491.

// full-width forms; the result counts only when it is plain
// ASCII once combining marks are dropped, so non-Latin scripts
// pass through unchanged.
let base: String = std::iter::once(c)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Low] Correctness: the ASCII gate runs before SPECIAL, so a precomposed letter whose base is a non-ASCII special keeps its accent: fold("ǣ") (æ + macron) stays ǣ, fold("ǿ") (ø + acute) stays ǿ, while æ→ae and ø→o.

Why this matters: the module promises Latin diacritics drop and ligatures/variants expand, so on a folding index a document spelled ǣ/ǿ is missed for an ae/o query (and the reverse) — a silent false negative.
Suggested fix: after stripping combining marks, map each remaining base char through the same SPECIAL/ASCII fold instead of requiring the whole base to be ASCII (e.g. fold the collected base with a reused scratch buffer).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant