Repository navigation
feat(executor): fulltext accent folding and wildcard terms (LABS-1460, ADR-0583) - #232
Conversation
| Target::Fuzzy { term, edits } => { | ||
| FieldMatch::Weighted(expand_fuzzy(term, *edits, stat)) | ||
| } | ||
| Target::Wildcard(pattern) => FieldMatch::Constant( |
There was a problem hiding this comment.
[Medium] Denial of service: uncapped wildcard expansion is re-materialized per clause and per field, so a short query forces memory proportional to clauses times vocabulary.
Why this matters: every wildcard clause clones each matching vocabulary term into its own HashSet (stat.df.keys()...cloned().collect()), so a query of N repeated * tokens allocates N copies of the index vocabulary; over Bolt/HTTP (16 MiB statement/message limit) a few KiB of * can become gigabytes and OOM the store, aborting every session on it.
Suggested fix: expand each distinct (field, pattern) once per scan and share the set across identical clauses (or test document tokens against the Pattern directly), and add a test that peak memory does not scale with clause count.
|
Security review (security-review skill) The diff adds
Security review: |
| fulltext::Analyzer::from_value(&eval(graph, input, expr, params, cypher_version)?) | ||
| }) | ||
| .transpose()?; | ||
| let analyzers = fulltext::Analyzers::new( |
There was a problem hiding this comment.
[Medium] Option handling: WITH ANALYZER NULL becomes an explicit Standard override instead of the documented "no override", so it bypasses a folding index's query-side folding.
Why this matters: Analyzer::from_value maps Value::Null to Analyzer::Standard (fulltext.rs:141-148 calls NULL CIP-225's "no override"), and the new Analyzers::new(index, Some(Standard)) path then skips index.query_default(); on a standard-folding index an accented term such as Eifrém misses with WITH ANALYZER NULL but matches when the clause omits it.
Suggested fix: keep NULL as absent — map Value::Null to None before from_value (or return Option<Analyzer> from the override resolution) so Analyzers::new applies query_default(); add a folding-index test for WITH ANALYZER NULL.
| for term in tokenize(term) { | ||
| let edits = suffix | ||
| .fuzzy | ||
| .map_or(0, |fuzz| fuzz.edits_for(term.chars().count())); |
There was a problem hiding this comment.
[Low] Option handling: suffix.fuzzy.map_or(0, ...) fabricates an exact-match default for a value the enclosing else if suffix.fuzzy.is_some() already guarantees is Some.
Why this matters: the None arm is unreachable today, but if that guard changes every fuzzy clause silently degrades to distance 0 instead of failing — the repo bans fabricated defaults over invariant-bound values.
Suggested fix: bind the invariant once (let Some(fuzz) = suffix.fuzzy else { return Err(malformed(text, "a fuzzy suffix is required here")); };) and call fuzz.edits_for(...) in the loop.
|
Rust review (rust-reviewer skill, full review) Full-diff review of the executor-side fulltext change:
Rust review: ✅ no issues found |
| pub(crate) fn index_analyzer( | ||
| options: &BTreeMap<String, Value>, | ||
| index_name: &str, | ||
| ) -> Result<Analyzer> { |
There was a problem hiding this comment.
[Medium] structure (std-6): This is the fulltext scan's stored-config read — it turns a stored indexConfig value into a query-time Analyzer and raises a query-time drift error — but it lives in the schema-DDL options module,
Suggested fix: Move the stored read beside the analyzer policy (e.g. fulltext::Analyzer::from_index_options(options, index_name)) and give the scan one resolution entry point — resolve_fulltext_index/Analyzers::for_index(options, query_override) — that both exec::search and exec::procedures call, mirrori
finding F-1-1
| Error::execution(format!( | ||
| "InvalidIndexDefinition: fulltext index {index_name} has an invalid indexConfig option" | ||
| )) | ||
| }; |
There was a problem hiding this comment.
[Medium] comparison (flag-canonical-dup): The stored-option drift contract already exists as exec::vector::invalid_index_option (vector.rs:172-176), whose doc claims it is "One drift contract for every invalid stored index option" and whose s
Suggested fix: Hoist one index-kind-aware drift constructor into this shared options module (the layer exec::vector already imports) and call it from both stored_config and index_analyzer; drop the local closure.
finding F-1-10
| .as_ref() | ||
| .map(|expr| { | ||
| fulltext::Analyzer::from_value(&eval(graph, input, expr, params, cypher_version)?) | ||
| }) |
There was a problem hiding this comment.
[Medium] comparison (std-5): WITH ANALYZER's two spellings of "no analyzer named" now diverge: an absent clause reaches Analyzers::new as None and follows the index (Analyzer::query_default), while an explicit NULL is lowered to
Suggested fix: Give "no analyzer named" one representation: resolve the evaluated WITH ANALYZER to Option (NULL -> None) if NULL is no-override, or to a named three-state enum if NULL must mean standard; fix the comments to state the chosen policy and pin it with a WITH ANALYZER null case on a folding in
finding F-1-11
| c, | ||
| &format!( | ||
| "MATCH (p:{label}) \ | ||
| SEARCH p IN (FULLTEXT INDEX idx{label} FOR '{query}' LIMIT 50) \ |
There was a problem hiding this comment.
[Medium] comparison (flag-canonical-dup): The speaker corpus re-copies the people-store drivers: speaker_search is search_names with only the label, index and LIMIT (Person/personText/10) substituted, and speaker_procedure is procedure_names
Suggested fix: Parameterize the corpus (label, index, limit) in one SEARCH-clause driver and one queryNodes driver, make the people and speaker call sites thin wrappers, and keep speakers() as the one dual-entry-point assertion.
finding F-1-12
| pub(super) fn parse(term: &str) -> Option<Self> { | ||
| let mut parts = Vec::new(); | ||
| let mut wild = false; | ||
| let mut chars = term.chars().flat_map(char::to_lowercase); |
There was a problem hiding this comment.
[Medium] comparison (std-6): Pattern::parse is matched against stored tokens, which tokenize lowercases with str::to_lowercase (fulltext.rs:246), but parse lowercases per char with char::to_lowercase. The two primitives differ wh
Suggested fix: Lowercase the term once with the stored side's primitive (let lowered = term.to_lowercase(); then iterate lowered.chars()) so pattern atoms and the tokens they match come from one lowercasing rule; leave the wildcard's no-stemming/no-folding policy unchanged.
finding F-1-13
| if quoted && matches!(c, '*' | '?') { | ||
| current.push('\\'); | ||
| } | ||
| current.push(c); |
There was a problem hiding this comment.
[Medium] branching (std-2): Quotedness is carried out of the tokenizer as an injected backslash character that Pattern::parse later un-escapes, so the quoting policy is encoded in the word text instead of in the word itself: a
Suggested fix: Yield the word with its quoting as data (a small Word { text, quoted }), and let push_text skip only the wildcard branch for a quoted word; delete the escape injection and its branch.
finding F-1-5
| Error::execution(format!( | ||
| "InvalidIndexDefinition: fulltext index {index_name} has an invalid indexConfig option" | ||
| )) | ||
| }; |
There was a problem hiding this comment.
[Medium] boundaries (std-6): The new fulltext read path spells its own drift rejection, while exec::vector::invalid_index_option is documented as "One drift contract for every invalid stored index option" and renders the identi
Suggested fix: Promote the drift constructor to the shared scan module (exec::candidates, already the home of resolve_search_index), parameterized by kind and index name, and call it from both exec::vector::stored_config and index_analyzer; the local closure then disappears.
finding F-1-6
| crate::exec::ddl_options::index_analyzer(&options, &index_name)?, | ||
| None, | ||
| ); | ||
| let produced = fulltext::search_all(graph, &detail, &query, analyzers)? |
There was a problem hiding this comment.
[Medium] boundaries (std-6): The fulltext catalog phase — resolve_search_index (lines 650-651), the stored-analyzer read, Analyzers::new, EntityKind::from_relationship_flag, and candidate materialization — is orchestrated a
Suggested fix: Extract the shared kernel into exec::fulltext (mirroring vector_index_candidates): resolve the index, build the Analyzers from the stored options plus an optional override, and return (EntityKind, Vec<(Value, f64)>); both fulltext_index_candidates and exec_fulltext_query_call call it.
finding F-1-7
| "MATCH (p:{label}) \ | ||
| SEARCH p IN (FULLTEXT INDEX idx{label} FOR '{query}' LIMIT 50) \ | ||
| SCORE AS score RETURN p.name AS name, score ORDER BY score DESC, name ASC" | ||
| ), |
There was a problem hiding this comment.
[Medium] boundaries (std-6): speaker_search (line 534) re-implements the pre-existing search_names SEARCH template, differing only in the label/index and LIMIT; a second copy of one query shape is already drifting (LIMIT 10 v
Suggested fix: Parameterize the existing helper — search_names(c, label, index, limit, query) — and call it from both the speaker/folding corpus and the fuzzy/boost tests instead of adding a second builder.
finding F-1-8
| c, | ||
| &format!( | ||
| "CALL db.index.fulltext.queryNodes('idx{label}', '{query}') \ | ||
| YIELD node, score RETURN node.name AS name, score ORDER BY score DESC, name ASC" |
There was a problem hiding this comment.
[Medium] boundaries (std-6): speaker_procedure is the scored form of the pre-existing procedure_names over the same queryNodes template; only the label/index and the score projection differ, so the file now spells one CALL
Suggested fix: Fold both into one scored helper parameterized by label/index, and project the names where the callers need them.
finding F-1-9
|
Stellar review (stellar-review skill) Round 8 · head Still open from earlier rounds
Stellar review: |
3104db1 to
f917343
Compare
| /// and, by ruling, wildcard terms). | ||
| pub(crate) fn folds(self) -> bool { | ||
| matches!(self, Self::StandardFolding) | ||
| } |
There was a problem hiding this comment.
[Medium] structure (std-6): The analyzer exposes its folding policy as a bare flag whose only consumer is the query parser (query.rs:207), so the parser reconstructs the analyzer's term preparation itself — `fold(&term.to_lowerc
Suggested fix: Give Analyzer one preparation entry point (e.g. fn term_text(self, term: &str) -> String that lowercases and folds when the variant folds) and have query.rs call it unconditionally, deleting the folds() branch and the parser's fold import; the flag then disappears from the analyzer's surfa
finding F-2-1
| /// and, by ruling, wildcard terms). | ||
| pub(crate) fn folds(self) -> bool { | ||
| matches!(self, Self::StandardFolding) | ||
| } |
There was a problem hiding this comment.
[Medium] branching (std-0): folds is the fourth independent projection of Analyzer's three variants — analyze, stored_tokens and query_default each re-decide folding/stop-words/stemming in their own match — and it ex
Suggested fix: Make the behaviour data instead of four matches: Analyzer carries the flags its 45-row ANALYZERS table already implies (fold / stop_words / stems) behind one token pipeline, so folds becomes a field read, analyze/stored_tokens/query_default stop re-deciding the same policy, and the
finding F-2-2
| )) | ||
| }; | ||
| let config = index_config(options).map_err(|_| drift())?; | ||
| match config.and_then(|config| config.get(KEY_FULLTEXT_ANALYZER)) { |
There was a problem hiding this comment.
[Medium] Error handling: index_analyzer reads fulltext.analyzer with an exact-case lookup while create-time validation admits the key in any case, so an accepted FULLTEXT.ANALYZER config is silently ignored and the index behaves as standard.
Why this matters: validate_config_family lowercases indexConfig keys for admission (ddl_options.rs:534), but apply_fulltext_defaults then inserts the canonical lowercase fulltext.analyzer entry, so this read finds the default instead of the user's standard-folding and stored tokens are never folded — a silent wrong-result path for a DDL statement the engine accepted.
Suggested fix: normalize the keys once at create, or look this key up case-insensitively (for example config.iter().find(|(key, _)| key.eq_ignore_ascii_case(KEY_FULLTEXT_ANALYZER)).map(|(_, value)| value)).
| @@ -117,14 +116,19 @@ fn fulltext_index_candidates( | |||
| // FulltextSearchAcceptance [57]); an absent slot means the standard | |||
There was a problem hiding this comment.
Low — Docs: The comment still says an absent WITH ANALYZER slot means the standard analyzer.
Why this matters: Analyzers::new(index, None) now resolves to Analyzer::query_default(), so a standard-folding index folds query terms even without WITH ANALYZER; the exec/fulltext.rs module header carries the same drift, still describing "the two differentiated search analyzers (standard, english)" though standard-folding is now a third.
Suggested fix: Say "absent means the index's query default (its folding, not its stemming)" here, and update the module header to name three analyzers.
| )) | ||
| }; | ||
| let config = index_config(options).map_err(|_| drift())?; | ||
| match config.and_then(|config| config.get(KEY_FULLTEXT_ANALYZER)) { |
There was a problem hiding this comment.
Low — Silent fallback: The stored indexConfig read matches fulltext.analyzer by exact case, but validate_config_family admits the key in any casing.
Why this matters: an index created with the key spelled FULLTEXT.ANALYZER passes validation, then apply_fulltext_defaults adds the canonical fulltext.analyzer default next to it, so this lookup silently returns the default and the configured analyzer (for example standard-folding) is never applied to queries.
Suggested fix: Look the key up case-insensitively (for example config.iter().find(|(key, _)| key.eq_ignore_ascii_case(KEY_FULLTEXT_ANALYZER))), or normalise the stored config keys at the create boundary.
b36a6a3 to
212ebb1
Compare
f917343 to
81422a0
Compare
| Target::Fuzzy { term, edits } => { | ||
| FieldMatch::Weighted(expand_fuzzy(term, *edits, stat)) | ||
| } | ||
| Target::Wildcard(pattern) => FieldMatch::Constant( |
There was a problem hiding this comment.
[Medium] Denial of service (resource exhaustion): each wildcard clause clones its whole matching field vocabulary into a FieldMatch::Constant, so query memory grows as (wildcard clauses × distinct indexed terms).
Why this matters: the fulltext query text is caller-controlled and the 4 MiB statement cap is no bound here — * * * … costs two bytes per clause, and every clause independently retains a full vocabulary copy, so a ~10 KB read-only query (SEARCH … FULLTEXT INDEX or db.index.fulltext.queryNodes) against a 100k-term index retains hundreds of millions of cloned strings and OOMs the process. Pattern::matches also costs O(pattern × term) per vocabulary term, so one long ? pattern adds a CPU multiplier on top.
Suggested fix: expand each distinct pattern once per scan and share the set across clauses, and bound total expanded terms (or wildcard clause count) per query with a typed error rather than silent truncation, keeping ADR-0583's no-silent-drop constraint.
| } | ||
| } | ||
| words.push(word.to_string()); | ||
| words.push(word); |
There was a problem hiding this comment.
[Medium] structure (std-6): This round's rewrite moved the detached-suffix rewrite into the flush step, so emitting a word also reaches back and mutates the previously emitted one — and to decide whether it may, the scanner ha
Suggested fix: Make words one stage: scan to whole words (escapes and quotes only) and return them, then attach a detached ~/^ word to its predecessor in a separate pass — e.g. fn attach_suffixes(words: &mut Vec<String>) called from parse next to Parser::word's operator folding — so the emission step o
finding F-4-1
| } else { | ||
| current.push(other); | ||
| } | ||
| } |
There was a problem hiding this comment.
[Medium] branching (q-missing-helper): The escape arm re-decides the word-boundary policy the main loop owns eight lines below (} else if c.is_whitespace() || matches!(c, '(' | ')') {): the separator test now exists in two places, and an
Suggested fix: Peel the escape once and hand every character — escaped or not — to the same disposition (flush on a separator, else append), so the separator test exists in one place; if the scan keeps its shape, extract fn is_separator(c: char) -> bool and call it at both sites.
finding F-4-2
| Some(next @ ('*' | '?' | '\\')) => { | ||
| current.push('\\'); | ||
| current.push(next); | ||
| } |
There was a problem hiding this comment.
[Medium] branching (std-0): The escape match spells "this character is a literal in pattern text" by hand in each arm — an escaped metacharacter as a backslash push plus the char, an escaped backslash as a literal two-backslash
Suggested fix: Put the one way to append a literal character to pattern text in wildcard (e.g. Pattern::push_literal(&mut String, char), escaping the metacharacters) and route the escape arms through it: the metachar and escaped-quote arms collapse into one push_literal(next) arm, and the trailing-escape and
finding F-4-3
| } else { | ||
| current.push(other); | ||
| } | ||
| } |
There was a problem hiding this comment.
[Medium] boundaries (std-5): The arm above promises any escaped character "stands for itself", but it consumes the backslash and hand the bare character on: Parser::word still splits an escaped a\:b at the colon, flush/`spl
Suggested fix: Carry the word as typed text (chars with a literal flag, or a Word value) and decode escapes once at the boundary so Parser::word and split_suffix consult literalness instead of re-splitting a string; alternatively escape every downstream metacharacter at this one site. Extend the new escape t
finding F-4-4
| current.push(other); | ||
| } | ||
| } | ||
| None => current.push_str("\\\\"), |
There was a problem hiding this comment.
[Medium] boundaries (std-5): An escape with no following character is silently repaired: instead of the typed malformed syntax error every other malformed suffix raises, the lexer fabricates a literal backslash, which `tokenize
Suggested fix: Return Result<Vec<String>> from words (its only caller parse is already fallible) and raise malformed(query, ...) for a dangling escape; change the test's "does not panic" case to assert the typed syntax error.
finding F-4-5
| pub(super) fn parse(term: &str) -> Option<Self> { | ||
| let mut parts = Vec::new(); | ||
| let mut wild = false; | ||
| let mut chars = term.chars().flat_map(char::to_lowercase); |
There was a problem hiding this comment.
[Low] Correctness: Pattern::parse lowercases with char::to_lowercase while stored tokens are lowercased by str::to_lowercase in tokenize; the two disagree on Greek final sigma.
Why this matters: "ΑΣ" is stored as ας, but the pattern ΑΣ* lowercases to ασ*, so on a non-folding index that wildcard never matches its own stored token while the plain term ΑΣ does.
Suggested fix: lowercase the term as a string first (let lowered = term.to_lowercase(); then iterate lowered.chars()), so the wildcard path shares the tokenizer's lowercase policy.
| fulltext::Analyzer::from_value(&eval(graph, input, expr, params, cypher_version)?) | ||
| }) | ||
| .transpose()?; | ||
| let analyzers = fulltext::Analyzers::new( |
There was a problem hiding this comment.
[Medium] Option handling: an explicit WITH ANALYZER NULL evaluates to Analyzer::Standard and is passed as Some(..), so it bypasses query_default() and turns off the index's folding.
Why this matters: Analyzer::from_value documents NULL as CIP-225's "no override", yet on a standard-folding index FOR 'eifrem' WITH ANALYZER NULL now misses Eifrém while omitting WITH ANALYZER finds it — the two forms silently diverge, and the comment just above (line 116) still says "an absent slot means the standard analyzer".
Suggested fix: collapse an evaluated Value::Null to None before Analyzers::new (e.g. match the evaluated value: Value::Null => None, other => Some(Analyzer::from_value(&other)?)), and add a folding-index test for WITH ANALYZER NULL.
81422a0 to
d8d0ca9
Compare
| Target::Fuzzy { term, edits } => { | ||
| FieldMatch::Weighted(expand_fuzzy(term, *edits, stat)) | ||
| } | ||
| Target::Wildcard(pattern) => FieldMatch::Constant( |
There was a problem hiding this comment.
[Medium] Denial of service: Every wildcard clause independently clones all matching vocabulary terms into its own HashSet, uncapped and uncharged by the ADR-0463 allocation budget.
Why this matters: the SEARCH query (and the fulltext procedures) is client-controlled text on every surface, so a ~2 KB parameter of 1,000 * clauses retains 1,000 × |vocabulary| owned strings — gigabytes on a 100k-term index, before scoring starts — letting a tiny query OOM the process past the configured cap.
Suggested fix: resolve each distinct (pattern, field) expansion once per scan and charge the retained set via ExecGraph::check_expression_allocation; sharing does not truncate hits, so the ADR's no-cap parity goal still holds.
| last.push_str(word); | ||
| if let Some(last) = out.last_mut().filter(|last| operator(last).is_none()) { | ||
| last.push_str(&word); | ||
| continue; |
There was a problem hiding this comment.
[Medium] branching (std-2): The suffix grammar has no single owner: this new glue stage decides "detached suffix" by spelling the marker set as starts_with(['~', '^']), while split_suffix (line 287) independently re-lists th
Suggested fix: Put the marker vocabulary beside the Suffix type and have both stages ask it — e.g. impl Suffix { fn begins(word: &str) -> bool; fn find(text: &str) -> Option<usize> }, or one const SUFFIX_MARKERS: [char; 2] consulted by attach_suffixes and by split_suffix's find_unescaped call — so the
finding F-5-1
| } | ||
| } | ||
| flush(&mut current); | ||
| Ok(words) |
There was a problem hiding this comment.
[Medium] boundaries (std-5): quoted is scanned state that is never inspected at end of input: an unterminated quote is silently flushed as if it had closed, while this same Result-returning lexer raises a typed malformed er
Suggested fix: Decide the invariant and apply it once at EOF: raise malformed(query, "an unterminated quote") when quoted is still true after the loop (live's Lucene lexer rejects it), or, if accepting it is deliberate, state that shortcut in the module-header list and pin it with a test; either way the rule l
finding F-5-2
| // quoted `f:x` has no property either. | ||
| for query in [r"a\:b", r#""f:x""#] { | ||
| let parsed = clauses(query); | ||
| assert!(parsed.iter().all(|c| c.property.is_none()), "{query}"); |
There was a problem hiding this comment.
[Medium] boundaries (std-5): Both assertions (here and the lamp\~1 check below) are satisfiable by the broken parse the test exists to catch: the escaped \:/\~ reaches tokenize, which splits on the backslash, so a\:b yi
Suggested fix: Assert the positive contract instead of the absence of two wrong shapes: assert_eq!(parsed.len(), 1) plus assert_eq!(term(&parsed[0]), "a:b"), and the equivalent single-term/Fuzzy assertions for lamp\~1, so the test fails until escaped characters are actually carried inside one word.
finding F-5-3
| let (property, text) = match find_unescaped(word, &[':']) { | ||
| Some(at) => (Some(&word[..at]), &word[at + 1..]), | ||
| None => (None, word), | ||
| }; |
There was a problem hiding this comment.
[Medium] boundaries (std-5): The changed split now finds the delimiter past escaped colons, but the property half keeps those escapes: a\:b:term yields a clause property of a\:b, which resolve_clause compares raw against fi
Suggested fix: Give both halves of the split the same decoding: run the property slice through the same escape pass the term text uses (or resolve it to a typed property name that owns the decoding) so a\:b:term names the field a:b, rather than emitting a property string no field can ever equal.
finding F-5-4
212ebb1 to
f258817
Compare
|
Deferred review findings → LABS-1491 (review-tasks, Agent Backlog) with a |
d8d0ca9 to
bf341cc
Compare
| for c in token.chars() { | ||
| if c.is_ascii() { | ||
| out.push(c); | ||
| } else if unicode_normalization::char::is_combining_mark(c) { |
There was a problem hiding this comment.
[Low] Correctness: dropping standalone combining marks makes the fold depend on the Unicode normal form of non-Latin input.
Why this matters: this branch drops every standalone combining mark, so NFD Greek α + tonos folds to α while the NFC form ά passes through unchanged. Two canonically equivalent spellings of the same word then produce different stored tokens, contradicting the documented "Greek and Cyrillic are untouched" guarantee (the NFKD branch comment below makes the same claim).
Suggested fix: drop the mark only when the base it follows was actually folded to ASCII (track whether the previous output came from an ASCII fold); otherwise keep the original character with its marks.
| pub(super) fn parse(term: &str) -> Option<Self> { | ||
| let mut parts = Vec::new(); | ||
| let mut wild = false; | ||
| let mut chars = term.chars().flat_map(char::to_lowercase); |
There was a problem hiding this comment.
[Low] Idiomatic Rust: wildcard terms are lowercased per char while stored tokens are lowercased with str::to_lowercase, so the two disagree on the Greek final sigma.
Why this matters: str::to_lowercase applies the context-sensitive final-sigma rule (a capital Σ at a word end becomes ς), but char::to_lowercase always yields σ. The stored token for ΣΟΦΟΣ is σοφος, while Pattern::parse("ΣΟΦΟΣ*") builds σοφοσ*, so the prefix wildcard never matches the word that the exact term ΣΟΦΟΣ matches.
Suggested fix: lowercase the whole term first (let lowered = term.to_lowercase();) and iterate lowered.chars(); escapes, * and ? are unaffected by case mapping.
| .analyzer | ||
| .as_ref() | ||
| .map(|expr| { | ||
| fulltext::Analyzer::from_value(&eval(graph, input, expr, params, cypher_version)?) |
There was a problem hiding this comment.
[Medium] Option handling: WITH ANALYZER NULL is silently turned into an explicit Standard override, so it stops following the index folding.
Why this matters: Analyzer::from_value documents NULL as "no override" and returns Standard; wrapping that in Some(...) makes Analyzers::new treat it as an override. On a standard-folding index, FOR 'eifrem' WITH ANALYZER NULL no longer folds and misses Eifrém, while the same query without the clause finds it. The comment above ("an absent slot means the standard analyzer") is also stale, since an absent slot now means the index default.
Suggested fix: make the override Option<Analyzer> end to end (NULL -> None, so Analyzers::new falls back to index.query_default()), or map NULL to the index analyzer here, and update the comments.
Spec-harness coverage — 🟡
Rerun |
|
Loadtest baselines comparison
Loadtest baseline comparisonFrozen baselines vs current reports; bands match the Status: warning Pairs: fraud-loadtest: compared · identity-loadtest: compared (failing rows) fraud-loadtest
Per-use-case p99 latency
identity-loadtest
Per-use-case p99 latency
|
|
Memory controls and workspace memory calibration
(comparison document unavailable) |
|
Memory shape
Memory shapeStatus: ok All memory shapes held. 🟢 baseGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the base shape is loaded to 0 nodes and 0 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 0.00 MB – 0.00 MB, disk 0.00 MB – 0.00 MB, attributed RSS 0.00 MB – 0.00 MB, ratio 0.000x–0.000x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 0.00 MB, disk 0.00 MB, attributed RSS 0.00 MB, ratio 0.000x. Report: logical, disk, attributed RSS, and the ratio held. flatGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the flat shape is loaded to 120000 nodes and 0 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 26.02 MB – 26.02 MB, disk 1.95 MB – 1.95 MB, attributed RSS 36.49 MB – 36.93 MB, ratio 1.402x–1.419x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 26.02 MB, disk 1.26 MB, attributed RSS 37.73 MB, ratio 1.450x. Report: logical, disk, attributed RSS, and the ratio held. fanoutGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the fanout shape is loaded to 52016 nodes and 52000 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 28.62 MB – 28.62 MB, disk 1.75 MB – 1.75 MB, attributed RSS 34.93 MB – 36.61 MB, ratio 1.220x–1.279x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 27.43 MB, disk 1.10 MB, attributed RSS 35.94 MB, ratio 1.310x. Report: logical, disk, attributed RSS, and the ratio held. ringGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the ring shape is loaded to 46000 nodes and 46000 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 29.99 MB – 29.99 MB, disk 1.77 MB – 1.77 MB, attributed RSS 35.30 MB – 35.89 MB, ratio 1.177x–1.197x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 29.99 MB, disk 1.20 MB, attributed RSS 35.68 MB, ratio 1.190x. Report: logical, disk, attributed RSS, and the ratio held. dense ringGiven: an empty relate serve, 1 integer properties on every node, text indexes 0, point indexes 0, range indexes 0, and a 32.00 MB logical ceiling. When: the dense ring shape is loaded to 5000 nodes and 100000 relationships, then checkpointed. The child RSS is sampled. Then: the database holds those counts. The load finishes under the ceiling. Logical bytes, disk, attributed RSS, and the ratio stay inside the frozen boundaries. Baseline: logical 31.41 MB – 31.41 MB, disk 1.62 MB – 1.62 MB, attributed RSS 31.27 MB – 32.46 MB, ratio 0.996x–1.034x. A reading may sit 5% outside the ratio, and 15% or 1.00 MB outside the sizes. Validated: logical 31.11 MB, disk 0.73 MB, attributed RSS 31.87 MB, ratio 1.024x. Report: logical, disk, attributed RSS, and the ratio held. |
bf341cc to
20f132c
Compare
|
Rebased onto current main ( |
Oplane Security ReviewOplane identified security requirements relevant to this PR and checked the implementation against them, finding that 1 requirement needs attention. Push changes to rerun the review, update a status with a justification if needed, or review the generated threat model.
Change comment settings · Set up Oplane in your agent · Workflow guidance |
A standard-folding index now folds Latin diacritics and ligatures in its stored tokens and in analyzed query terms (the index's configured analyzer is read). Wildcard terms (term*, te?m, *term, field:term*, *:*) expand over each field's vocabulary into the any-of group the fuzzy path uses and score a constant 1.0 per matching field times the boost, with no expansion cap. Under a folding analyzer the literal parts of a wildcard are folded like stored tokens (a deliberate deviation from live Neo4j, which leaves them unfolded); fuzzy terms stay lowercased only, and escaped or quoted wildcard characters are plain text. Calibrated against Neo4j 2026.09-enterprise. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
20f132c to
3005586
Compare
| Target::Fuzzy { term, edits } => { | ||
| FieldMatch::Weighted(expand_fuzzy(term, *edits, stat)) | ||
| } | ||
| Target::Wildcard(pattern) => FieldMatch::Constant( |
There was a problem hiding this comment.
[Medium] Denial of service: Uncapped wildcard expansion is rebuilt per clause, so query-time memory grows as clauses × field vocabulary.
Why this matters: each wildcard clause clones every matching term into its own HashSet<String> for every indexed field; clause count is user-controlled (a 4 MiB statement or a large db.index.fulltext.queryNodes parameter holds ~2M * clauses), so a long query over a small index exhausts RSS and OOMs the process, bypassing the existing budget seam (ADR-0463) and fuzzy's 50-term cap.
Suggested fix: bound the expansion per field as MAX_FUZZY_EXPANSIONS does, or share one expansion per (pattern, field) and charge it to the statement budget.
|
Deferred security-review finding → LABS-1498 (review-tasks, Agent Backlog, |
| // full-width forms; the result counts only when it is plain | ||
| // ASCII once combining marks are dropped, so non-Latin scripts | ||
| // pass through unchanged. | ||
| let base: String = std::iter::once(c) |
There was a problem hiding this comment.
[Low] Correctness: the ASCII gate runs before SPECIAL, so a precomposed letter whose base is a non-ASCII special keeps its accent: fold("ǣ") (æ + macron) stays ǣ, fold("ǿ") (ø + acute) stays ǿ, while æ→ae and ø→o.
Why this matters: the module promises Latin diacritics drop and ligatures/variants expand, so on a folding index a document spelled ǣ/ǿ is missed for an ae/o query (and the reverse) — a silent false negative.
Suggested fix: after stripping combining marks, map each remaining base char through the same SPECIAL/ASCII fold instead of requiring the whole base to be ASCII (e.g. fold the collected base with a reused scratch buffer).
Summary
New feature (its stacked base #227 has merged): a
standard-foldingfulltext index now folds Latin diacritics, and the query syntax gains wildcard terms (term*,te?m,*term). Reported against the Aura speaker template:eifremmissedEifrém,andreas saterasmissedAndreas Satéras, and aKollegg*prefix matched nothing.Changes
standard-foldingis a real analyzer (ASCII folding of Latin diacritics, ligatures and letter variants; Greek and Cyrillic untouched). The index's configured analyzer is read, so stored tokens and analyzed query terms both fold.*any run,?one character, leading wildcard,field:term*,*/*:*match-all,\*escape, quoted wildcards are plain text. They expand over each field's vocabulary into the existing any-of group; a matching field scores a constant 1.0 times the boost; no expansion cap.Müll*,mull*,MÜLL*andEIFRÉ*all find Müller/Eifrém. This is a deliberate deviation from live Neo4j 2026.09, which leaves wildcards unfolded (recorded in ADR-0583).?counts one character of the folded token (stra?edoes not matchstrasse).eifrém~0still misses on a folding index.docs/reference/cypher-syntax.mdfulltext table and folding note updated.User-facing impact
Folding indexes match accent-insensitively;
*and?outside quotes in a fulltext query now mean wildcard (before they were separators).Example usage
Verification
cargo test -p relate-executor fulltext(42 passed),cargo test -p relate --test fulltext_search_semantics(19 passed; SEARCH clause and procedures)cargo fmt --all --check;rustup run 1.88 cargo clippy --workspace --all-targets --exclude relate-python -- -D warningsFixes LABS-1460.
🤖 Generated with Claude Code
Rebased onto current main after #227 landed. While rebasing, its head was found to silently revert main's #244 (
creation_failurewrapper in the constraint matrix) and #245 (the ddl test's constraint-equivalence token) — its merge-base predated those fixes, so the revert was textually clean. The commit itself touches only its own 10 files, so it was cherry-picked onto current main; set-equality now lists exactly those 10 files with no reverts.