[oratis]

PR #404 (merged as 5335218) replaced htmlToText in src/tools/web_fetch.ts and parseDuckDuckGo in src/tools/web_search.ts with linear single-pass converters. That closed a real CPU-exhaustion hole: 400 KB of unclosed tags took over a minute. A focused review of that rewrite then found text-quality regressions, and the fix round for them never landed. They affect every edition, including the local Mac web_fetch. Verified on main fa44994:

Input to htmlToText	main (fa44994)	Before #404	This PR
<p>if a < b then c</p>	if a	full text	if a < b then c
<pre>cout << x << endl;</pre>	cout	full text	cout << x << endl;
Price < 5 <script>var secret=1;</script> tail	Price var secret=1; tail (the script body is shown as text)	script dropped	Price < 5 tail
<p>a</p><style-guide>b</style-guide><p>c</p>	a (the rest is hidden)	a b c	a b c (line breaks as before #404)
<p>a</p><script-x>b</script-x><p>c</p>	a	a b c	a b c

Both converters still make one forward pass, and every new branch has timing tests on hostile input.

What changed, item by item

1. A stray < is text (4b2a0d2). Markup starts only at < followed by a letter, </ followed by a letter, <! or <?, as in a browser. Any other < is kept and the scan moves on one character. Real tags, doctypes, comments and <?xml …?> are found as before.

2. Element names match by the whole name (5ab9c9e). script, style and noscript must be followed by HTML whitespace, /, > or the end of the input, so <script-loader>, <style-guide> and ODF's <style:style> are ordinary tags. End tags use the same set. Line-break tags (p, div, li, …) still match by prefix on purpose: that only adds a line break, never hides text, and it is what the converter before feat(cloud): SSRF-guarded web_search / web_fetch for cloud chat, off by default (LISA_CLOUD_WEB_TOOLS=1) #404 did (<pre>, <link> and <track> also break the line), so ordinary pages convert the same way.

3. format=raw returns markup as served (9acde74). A bracket run is now split where its direction changes. Only a part that points one way is folded, so >< between adjacent tags and <> are left alone. Ordinary HTML, XML (RSS) and JSON come back unchanged.

4. NEL (b5b8a05). quoteUntrusted now escapes U+0085 the same way as U+2028 and U+2029 (JSON.stringify only escapes characters below U+0020). This covers the fence header, a non-URL source and the web_search no-results notice.

5. Markup cap and notice (9503503). HTML_TO_TEXT_MAX_INPUT goes from 256 KB to 2 MB. That is more than web_fetch ever reads (1.6 MB at max_chars 200000), so fetched markup is never cut a second time. 2 MB of hostile markup converts in about 100 ms, even on a heavily loaded machine. When HTML or XML markup was left unread (cut at the byte cap), the output now ends with [markup cut at N KB] instead of a false [truncated at N chars]. Text longer than max_chars still gets [truncated at N chars], and both notices appear when both happened. The review's 396 KB page with a large inline head now returns its whole article at max_chars 200000; before this PR it returned no article text.

6. Look-alikes (81ec814). The defang is now driven by a table of characters that read as angle brackets: ASCII, fullwidth and small < >, ‹ ›, both 〈 〉, ⟨ ⟩, ⧼ ⧽, ❬ ❭ ❮ ❯ ❰ ❱, ˂ ˃, U+1D236/7, ᐸ ᐳ, ᚲ and U+16F3F. ≪ ≫ ⟪ ⟫ ⪡ ⪢ count as two brackets each, and ⋘ ⋙ ⫷ ⫸ as three. A run that points one way is folded to square brackets when:

   • two of its brackets touch (nothing between them, or only default-ignorable or combining characters), or
   • it holds three or more brackets with only blank space between them (any Unicode space, tab, line break, or U+2800).

   Letters in the table (ᐸ ᐳ ᚲ) fold only at three, so a doubled syllable is left alone. Format characters are removed first, as before.

   • Defused: every shape the review listed (< < <…> > > with any whitespace, ᐸᐸᐸ, ⋘ ⋙, ⫷ ⫸, ‹‹‹, ⟨⟨⟨, ❮❮❮, ˂˂˂), plus 〈〈〈 and ≪<.
   • Not defused (stated in the code comment): a single <END-EXTERNAL-CONTENT>; two brackets separated by space, < <…> >; a lone ≪ or ⟪; « » and 《 》; ≮; entities that the HTML step does not decode (<); a change of direction.
   • Side effect: the space rule also folds a three-level e-mail quote, > > >.

   The fence format is unchanged.

7. Format-character removal is documented (f5a416d). The comment at INVISIBLE_FORMAT lists what removing \p{Cf} also takes out, format=raw included:

   • ZWJ in emoji sequences, and the tag characters of subdivision flags (England, Scotland and Wales become a plain black flag);
   • ZWNJ and ZWJ in Persian, Arabic and Indic text;
   • RTL and LTR marks, and bidi embeddings, overrides and isolates;
   • the word joiner, soft hyphen, BOM, Mongolian vowel separator and Arabic number signs;
   • the rest of the Cf category.

   Behaviour is unchanged. A test pins these side effects.

Comparison with the pre-#404 converter

I compared old_impl.ts from the review with this branch:

• Corpus. 124 documents: hand-written news, docs (code blocks, tables, lists, <details>), CJK (Chinese, Japanese ruby, Korean), entities, inline scripts and styles, RSS and Atom; real pages from node_modules (bignumber.js API docs, the Playwright trace viewer shell); LISA's own five HTML pages; and 108 Markdown docs from this repo rendered with md-render. 122 are byte-identical. The two that differ are explained below.
• DuckDuckGo. Both repo test fixtures and a realistic results page give identical results at limits 1, 3, 5, 10 and 20.
• Fuzz (the review's r_diff, 3000 documents per mode):
  • well-formed documents: 0 differ;
  • with stray < and >: in 0 documents does the new output lack a word the old one had (on main there were 62);
  • with <script-x> and <style-guide>: 0 such documents.
• The review's s_cases. Every regression case now matches the old output.
• format=raw. 15 of 18 sources come back exactly as served. The other three are:
  • two pages whose inline scripts contain >>, >>> or << (the fence rule folds these on purpose);
  • one page longer than max_chars, which only differs by its truncation notice.

Differences from before #404 that this PR keeps, and why

• RSS <![CDATA[<p>…</p>]]>, one of the two corpus differences. Before feat(cloud): SSRF-guarded web_search / web_fetch for cloud chat, off by default (LISA_CLOUD_WEB_TOOLS=1) #404, line-break tags were replaced in an earlier pass, so the CDATA paragraph was dropped. Now its text is shown, followed by a stray ]]>; main does the same. Copying the old multi-pass behaviour would lose text and need rescanning.
• onclick="if(qty>0&&qty<10){…}", the other corpus difference. Neither converter parses quoted attribute values, so both leak part of the attribute: before feat(cloud): SSRF-guarded web_search / web_fetch for cloud chat, off by default (LISA_CLOUD_WEB_TOOLS=1) #404 it was 0&&qty, now it is 0&&qty<10){add()}">. This comes from the item 1 rule.
• cat <<EOF inside <pre>. <E opens a tag, as it does in a browser, so cat < is kept and the rest up to the next > is not.
• From feat(cloud): SSRF-guarded web_search / web_fetch for cloud chat, off by default (LISA_CLOUD_WEB_TOOLS=1) #404, not changed here:
  • an unclosed comment, script, style or noscript hides the rest of the page, as in a browser;
  • an end tag with whitespace before its > (</script >, </STYLE\n>) closes the element. Before feat(cloud): SSRF-guarded web_search / web_fetch for cloud chat, off by default (LISA_CLOUD_WEB_TOOLS=1) #404 that element's body showed up as text.

Tests

Every new behaviour test fails on fa44994. Two kinds pass there by nature: the timing tests, and the item 7 test, which pins documented behaviour that this PR does not change. New in src/tools/web_fetch.test.ts:

• a < that cannot start markup is text and does not swallow what follows
• script, style and noscript are matched by their whole name, not a prefix
• format=raw returns ordinary HTML, XML and JSON exactly as served
• bracket-heavy text is defanged in one linear pass
• quoteUntrusted escapes NEL (U+0085) as well as U+2028 / U+2029
• everything web_fetch reads is converted: a large head no longer hides the article
• markup left unread is reported as such, not as a max_chars truncation
• a fence look-alike (…) is folded: 17 cases
• what the fold leaves as written, by design
• removing format characters also changes ordinary text, as documented

New in src/tools/web_search.test.ts: a NEL (U+0085) in the query is escaped like U+2028 / U+2029.

Two existing tests were updated:

• the linear-time htmlToText tests now run at the 2 MB cap (under 1 s each) and include four stray-< shapes and two prefix-name shapes;
• <> is no longer folded.

No test or probe makes a real request: ambientFetch, lookup and transport are stubbed, and the global fetch throws.

Gates

• npm run typecheck: pass
• npm run typecheck:client: pass
• npm run lint: pass (0 errors; 72 warnings, none in the files this PR touches)
• npm run format:check: pass
• npm test (temporary LISA_HOME): 3426 tests, 3425 pass, 0 fail, 1 skipped
• npm run check:api-contract: pass

🤖 Generated with Claude Code