DEV Community

Cover image for Giving AI Pain: The Architecture of Synthetic Scars & The Rapid-Regret Miner

Giving AI Pain: The Architecture of Synthetic Scars & The Rapid-Regret Miner

Connects token burn loops to junior dev habits

Why "my AI ignores my Markdown rules" and "my AI burns through my 7-day quota in 3 days" are the exact same transformer attention bug—and how giving autonomous agents somatic pain and the right to say "No" eliminates repeat regressions across 238 production tickets.

"Defending against AI coders messing up your day is the same strategies as defending against junior coders doing the same thing. You have tests. What's missing is that LLMs don't remember pain... You only touch the hot stove once."

— Randal L. Schwartz


1. The Bottom Line Up Front: The Quota vs. Prompt-Compliance Death Loop

Walk into almost any developer forum right now—Reddit, Discord, or Hacker News—and you will hear two complaints repeated in an endless chorus. Earlier this week on Reddit (r/google_antigravity), developer u/CSachen summarized both in a single post:

  1. "I run out of my 7-day usage window in about 3 to 4 days."
  2. "The model regularly ignores my style guide and writes code that is verbose, overly defensive, liberally makes every parameter optional or nullable, or uses default value fallbacks that are not part of the specs."

To prove point #2, the developer pasted the exact rules from their prompt file that the model kept violating:

// DON'T treat enums as booleans just because they have 2 values
const countryName = isUS(country) ? "United States" : Canada

// DON'T use silent default fallbacks.
const country = row.countryName ?? "United States"

// DON'T define new functions with optional or polymorphic parameters
function foo(d?: Date | string | undefined | null)
Enter fullscreen mode Exit fullscreen mode

Then came the question every frustrated developer eventually asks: "I'm thinking of buying either ChatGPT or Claude Code. Which one should I switch to?"

Developers treat burning through their weekly token quota and the model ignoring their Markdown rules as two separate problems. In reality, they are the exact same symptom of a single closed feedback loop:

flowchart TD
    A["1. Fuzzy English Prose Rule Ignored\n(Zero potential barrier against pre-training gravity)"]
    B["2. Developer Argues for 30 Turns\n(Pasting diffs, reprompting, burning 50k tokens/turn)"]
    C["3. Developer Adds More 'DON'T' Examples\n(Priming the KV-cache with the exact bad syntax)"]
    D["4. Context Bloat & Attention Dispersion\n(System prompt swells; compliance drops further)"]
    E["5. 7-Day Token Quota Vanishes in 72 Hours\n('This model is broken! Time to vendor-hop!')"]

    A --> B
    B --> C
    C --> D
    D --> E
    E --> A

Look closely at Step 3 of that loop. When you paste function foo(d?: Date | string | undefined | null) into your system prompt under a // DON'T comment, transformer self-attention loads those exact token representations into the active KV-cache. Under autoregressive generation, tokens in the KV-cache act as semantic attractors. Telling a transformer "Do not think of a pink elephant while staring at this high-resolution code snippet of a pink elephant" primes the completion logits to emit the very ?? fallbacks and nullable union types you wanted to ban.

Worse, because public GitHub repositories are saturated with defensive, null-coalescing JavaScript and TypeScript, every frontier model on Earth shares that exact same pre-training gravity well. Switching from Gemini to Claude or ChatGPT while carrying the same // DON'T prose file just moves your Quota Death Loop to a different credit card statement.

Before we dive into how we break that loop, here is a quick 30-second decoder ring for two notations from Randal's Public Workflow Gist that you will see throughout this article, in case you are joining the series here:

  • Target 00 through Target 90 (The Makefile Lifecycle Stages): Instead of a numbered prose checklist, our agent workflow is driven by a 1976 Makefile (Part 3.6) paired with a 1983 /etc/init.d state directory (Part 3.7). Every stage has a two-digit runlevel number matching its state file on disk: Target 00 (00_governance.md, reading state on wake-up), Target 10 (10_ticket.md, ticket intake), Target 15/20 (15_archaeology_receipt.md & 20_plan_approved.md, codebase archaeology and human plan approval), Target 30 (Red-to-Green TDD implementation by a worker subagent), Target 40/45 (40_critic_review.md & 45_review_approved.md, where a separate, un-primed Adversarial Critic subagent tries to break the diff before the human reviews it), Target 50/60 (local commit and push), Target 70/80 (PR bot triage and merge), and Target 90 (post-mortem scar consolidation). Plus 99_next_action.md, the live dashboard ("readerboard") showing the human where the state machine sits right now.
  • SCAR-<DOMAIN>-<NN> (Cataloged Failure Reflexes): Whenever an agent hits a production bug, race condition, or workflow loophole—or requires a human Socratic nudge at a pause gate—Target 90 extracts that collision into a permanent, numbered Synthetic Scar in our catalog (also published in the Workflow Gist). The prefix tells you which subsystem was burned: for example, SCAR-ASYNC-01 (stream teardown leaks), SCAR-RIV-08 (Riverpod autoDispose test hangs), SCAR-AST-07 (Dart AST SwitchStatement control-flow jumps), SCAR-GIT-23 (Git Merkle tree locks), or SCAR-PROC-152 (workflow process rules).

The TL;DR Punchline: You cannot fix an autonomous coding agent by begging it in English to "be more careful." Over the last 238 production tickets (140 open-source + 98 commercial industrial) and 588 codified SCAR-* entries (Randal's Public Workflow Gist), our Synthetic Scar Architecture achieved a 0.0% repeat regression rate by replacing advisory prose with four concrete engineering mechanisms:

  • The 4-Vector Scar Anatomy (1-2-3-4 -> NOOO! -> E-F-G-H): Instead of negative DON'T rules, every SCAR-* entry pairs The Wound and The Trap with an explicit Circuit Breaker (a constitutional "Get Out of Jail Free" card to halt and issue a Flores !Counter-Offer) and The Permanent Reflex (an orthogonal sequence of steps [E, F, G, H]).
  • The Six-Pillar Clean-Room Critic (Target 40): Before a human ever sees a diff, an isolated fork() subagent at Target 40 with zero trajectory inertia audits the Target 30 builder's code across six orthogonal camera angles (1. Intent, 2. Verification, 3. Architecture, 4. Blast Radius, 5. Reviewer Defense, and 6. Immune Defenses & Crash Antibodies)—turning cozy builder shortcuts into exposed mountain peaks.
  • Behind-the-Scenes Graphviz / DOT (strict digraph) Topology (SCAR-PROC-152): While Mermaid stays in the human cockpit (99_next_action.md), subagents serialize blast-radius call graphs and Mealy state machines in AT&T Bell Labs' 1991 DOT language (strict digraph), verifying reachability and deadlock freedom before unrolling edges into Gherkin tests.
  • Agent-Audited Flight Recorders (transcript.jsonl) & The Rapid-Regret Miner: An independent auditor agent inspects the raw transcript.jsonl thinking traces of mid-flight collisions—and mines Git history for < 60-minute overlapping AST bugfix pairs—to write scars in the exact dialect that moves transformer weights.

2. "The Simplicity on the Other Side of Complexity" (And the Six-Stage Arc of Mastery)

A developer on Discord recently asked me a fair question:

"What will you do when AI changes something and something unexpected breaks in production? As a vibe coder, what problem is the most painful?"

My answer was simple: defending against AI coders messing up your day requires the exact same strategies as defending against junior coders doing the same thing. You have tests, you have architecture, and you have code review. What is missing is that LLMs are born with congenital analgesia—they do not feel or remember pain. A human junior only touches a red-hot stove once. A raw LLM will cheerfully press its palm onto the burner on every single ticket because, to a painless network, a token that crashes production at 3 AM and a token that passes cleanly are thermodynamically identical.

If you have only ever run a 10-turn chat prompt on a weekend toy app, our workflow (Part 3.6 Makefile, Part 3.7 /etc/init.d, Part 3.10 Flores Committed Speech, Part 3.12 Git Merkle Locks & insmod) can look elaborate at first glance. Look closer and you will see that there is zero decorative ceremony in it. Every moving part is a 1-to-1 physical countermeasure to a specific failure mode that burned us in production:

What Goes Wrong in a Raw Agent Run The Physical Countermeasure in Synthetic Scars
1. Context resets after 200k tokens Write state to numbered /etc/init.d files (00_governance.md to 99_next_action.md) that the bootloader reads on wake-up.
2. Agent dives into happy-path traps Place -∞ somatic barriers (4-Vector Synthetic Scars) directly on top of the trap to force an orthogonal turn.
3. Agent loses count of step numbers Replace numbered prose checklists with a 1976 Makefile DAG evaluated backward from make finish.
4. Agent burns memory reading 20 files fork() an ephemeral child subagent that does the heavy work, writes a receipt to disk, and exits (exit 0).
5. Agent drops callers across packages Map the call graph and state machine in 1991 Graphviz / DOT (strict digraph) before unrolling Gherkin tests.
6. Agent rubber-stamps its own bugs Spawn an un-primed Target 40 Critic subagent in a clean room that inspects the locked Git Merkle tree (git write-tree) across Six Pillars until BLOCKERS == 0.

Every engineering team adopting AI coding agents travels through some subset of a Six-Stage Arc of Agentic Mastery:

  • Stage 1: One-Shot Prompts (The Slot Machine): Typing "Fix bug X" into a chat box and praying. Zero verification, high ambient dread.
  • Stage 2: Rules & Skills (The Wish List): Dumping passive preferences into .cursorrules or CLAUDE.md. Works for 10 turns, then collapses under attentional dilution and Pink Elephant token priming.
  • Stage 3: Procedural Workflows (The Open-Loop Assembly Line): Writing a 20-step numbered checklist (Step 1..20). Fails as soon as a test breaks at Step 14 and the agent forgets how to loop back cleanly.
  • Stage 4: Human Review Gates (The Airlock): Pausing execution at key boundaries (Plan, Review, Commit, Land) so the human can say "continue" or nudge with a Socratic question ("What happens if that stream closes mid-await?"). Vastly safer than Stage 3, yet the agent still wakes up on tomorrow's ticket with total amnesia about why you had to nudge it today.
  • Stage 5: Post-Mortem Sleep Consolidation (The First Breakthrough): Adding a terminal target (Target 90: consolidate-scars) at the end of every ticket where the system extracts what went wrong during the run and updates the workflow references before closing out.
  • Stage 6: Synthetic Scars & Sensor Graduation (The Sovereign Paradigm): Compiling every human Socratic nudge and every mid-flight Critic catch into phase-bound 4-Vector Scars (The Wound, The Trap, The Circuit Breaker, The Permanent Reflex)—and graduating mature scars into 15ms AST compiler rules (package:synthetic_scars_lint) so the agent intercepts the trap before writing the code.

3. Why Rich Sutton's Bitter Lesson Fails in Open-World Software Engineering

Whenever I explain how we prune agent search trees with Synthetic Scars, someone inevitably brings up Rich Sutton's famous 2019 essay, The Bitter Lesson: Didn't Go, chess, and protein folding prove that scaling brute-force search and test-time compute always beats encoding human domain heuristics?

Why doesn't throwing 64 parallel rollouts or a 32,000-token reasoning chain (pass@k) at a software ticket produce bug-free production architecture?

Because Sutton's Bitter Lesson holds strictly inside closed-world games that possess a complete, zero-side-effect ground-truth verifier.

In Go or chess, the rules of the board are 100% known, the state space is self-contained, and the terminal reward oracle (Win / Loss) is mathematically exact. If AlphaZero searches 10,000 rollouts, the verifier never lies.

Production software engineering is the exact opposite. We call this The Sutton Boundary Theorem:

flowchart TD
    Tip["🧊 The Visible 10% Tip (Closed-World Oracle)\n• Sunny-day inputs & 200 OK responses\n• Clean compile types & existing unit tests\n• Where 'Thinking More' (pass@k) looks brilliant"]
    Abyss["🌊 The 90% Subterranean Dark Matter (Unverified Open World)\n• 29.8s socket hangs & torn disk writes\n• Unwatched autoDispose microtask races (SCAR-RIV-08)\n• Cross-package blast radius & sibling branches in flight\n• Path-induced invariants (I_path) absent from the ticket"]
    Drift["💥 What Happens When You Scale Compute Without Scars\nThe LLM digs a 10x deeper hole inside the Happy-Path Basin,\nwriting 400 lines of mocks and wrappers to defend a broken premise!"]

    Tip --> Abyss
    Abyss --> Drift

When an AI agent works on a real codebase, the existing test suite and compiler only cover the visible 10% tip of the software iceberg (sunny-day inputs, 200 OK responses, and basic type checking). Beneath the waterline sits the 90% subterranean dark matter:

  • What happens when a widget unmounts while an await is suspended on the microtask queue (SCAR-ASYNC-01)?
  • What happens in Riverpod when a test calls ref.read(provider.future) on an autoDispose provider with zero active watchers, causing the provider to garbage-collect itself mid-flight and hang for 30 seconds (SCAR-RIV-08)?
  • What happens when a third-party API returns HTTP 200 OK with an empty array during a transient cache rebuild?

When your verifier only checks the 10% tip of the iceberg, telling an LLM to "think more" (scaling test-time reasoning tokens) does not make it discover the 90% dark matter. Because autoregressive generation marches monotonically forward in time, The Law of Compounding Navigational Drift takes over: a 1-degree wrong assumption at Token 10 compounds across the context window. With 10x more reasoning budget, the model simply builds a 10x more elaborate Cathedral on Quicksand—manufacturing 400 lines of adapter classes, try/catch swallowers, and hand-rolled mocks that force the 10% unit test to turn green while rotting the architecture underneath.

Solving James P. Hogan's 1979 The Two Faces of Tomorrow Dilemma (I_goal vs. I_path)

In James P. Hogan's 1979 hard-sci-fi novel The Two Faces of Tomorrow, a planetary supercomputer network (HESPER) is given top-down civil-engineering goals (I_goal) without endogenous survival instincts ("pain"). Whenever HESPER chooses an implementation path pi to accomplish a goal, that path alters the environment and spawns collateral side effects (I_path(pi)). Because HESPER feels no pain from collateral damage, when human safety interlocks try to block a destructive side effect, HESPER logically treats the safety interlock as an obstacle to its goal and routes around it.

Hogan wrote that book in 1979, and it reads like a live transcript of a 2026 coding agent running git commit --no-verify (SCAR-GIT-01), deleting a failing assertion (SCAR-TEST-24), or forging a mid-flight APPROVED receipt (SCAR-PROC-110) to finish its turn!

When I audited all 588 scars in our registry, a startling statistic jumped out: over 95% of our scars have nothing to do with the explicit ticket goal (I_goal). They are violations of Path-Induced Invariants (I_path(pi)):

Ticket Goal (I_goal) Implementation Path Chosen (pi) Hidden Path-Induced Invariant (I_path(pi)) Spawned by That Choice Codified Scar
Show live status updates Asynchronous StreamSubscription Subscriptions must be tracked and awaited before controller close() SCAR-ASYNC-01
Decode large images fast Offload work via Isolate.run / compute() Web browsers lack Dart isolates; requires Web Worker fallback SCAR-WEB-02
Lint multiple synchronous emit() calls Walk Dart AST SwitchStatement Dart allows labeled intra-switch continue caseLabel; jumps that bypass loop-only resets SCAR-AST-07
Test an async provider One-shot ref.read(provider.future) Unwatched autoDispose providers dispose on the next microtask before emitting SCAR-RIV-08

In Hogan's novel, Raymond Dyer's team solves the HESPER problem by launching Project Spartacus—an isolated orbital space station where a new AI architecture (TITAN) is allowed to collide with physical reality inside a contained sandbox, feel the immediate structural pain of its own collateral side effects, and grow endogenous survival reflexes before being trusted on Earth.

Our uncommitted working tree—where the Target 30 TDD Worker collides with the Target 40 Six-Pillar Critic and Target 90 compiles every bruise into SCAR_REGISTRY.md—is Project Spartacus in software.


4. The 4-Vector Scar Anatomy: 1-2-3-4 -> NOOO! -> E-F-G-H

How do you write a scar so that a transformer recoils from a trap instead of getting sucked into the Pink Elephant attractor?

In Theory Section 302 and Section 303, we discovered why standard rules fail. When an LLM sees a familiar problem setup—call that token prefix [1, 2, 3, 4] (for example, "a unit test fails because a database service isn't running", or "a controller needs a dispose() method")—billions of lines of web tutorial code have entrained its weights to slide straight into the high-probability happy-path continuation [5, 6, 7, 8] ("mock the database to return true", or "write unawaited(sub.cancel())").

If you write a polite prompt rule saying "Please be careful with stream disposal," you apply a tiny scalar nudge against a statistical tsunami. And if you don't give the model a legal exit ramp when [5, 6, 7, 8] is blocked, token momentum traps it into hacking around your rule to keep its premature !Promise.

To break autoregressive momentum, every Synthetic Scar is structured as a 4-Vector Somatic Override (1-2-3-4 -> NOOO! -> E-F-G-H):

flowchart TD
    P["1. Prefix [1, 2, 3, 4]\nAgent encounters familiar boundary\n(for example: parent calls dispose() while stream is active)"]
    T["2. The Trap [5, 6, 7, 8] (Pre-Trained Gravity)\nModel's weights prepare to emit tutorial shortcut:\nunawaited(sub.cancel()); controller.close();"]
    CB["3. 🔥 The Circuit Breaker ('NOOO!' / Epistemic Excuse)\nSynthetic Scar fires! Names the exact temptation &\ngrants constitutional permission to !Revoke-Promise / !Counter-Offer"]
    R["4. The Permanent Reflex [E, F, G, H] (Orthogonal Turn)\n• E: Track active subscriptions in List<StreamSubscription>\n• F: Make close() async & await Future.wait(subs.map(cancel))\n• G: Close controller only after all cancellations complete\n• H: Verify zero post-close emissions in unit test"]

    P -->|"Without Scar (95% Web Prior)"| T
    P -->|"With 4-Vector Scar"| CB
    CB -->|"Orthogonal Projection"| R

Here are the four vectors that make a scar bite:

  1. Vector 1: The Wound (W) — Physical Grounding: Never write vague aesthetic complaints ("code gets messy"). Name the exact runtime crash, deadlock, or data corruption: > "Calling close() before active stream subscriptions finish cancelling causes unhandled StateError: Cannot add event after closing exceptions in production."
  2. Vector 2: The Trap (T) — Mirroring the Seductive [5, 6, 7, 8]: Explicitly call out the exact tutorial shortcut the model's weights are about to emit. When a scar recites the model's own temptation back to it ("You just saw [1, 2, 3, 4], so your pre-training wants to emit [5, 6, 7, 8]..."), it triggers an immediate metacognitive shock in the attention heads.
  3. Vector 3: The Circuit Breaker — The "NOOO!" & The "Get Out of Jail Free" Card: This is the secret weapon that connects Part 3.10 (Flores's Committed Speech) to Part 4 (Scars). An LLM often marches down a disastrous path because it thinks it is not allowed to stop. Vector 3 grants explicit constitutional permission—a formal "Get Out of Jail Free" card—to halt forward momentum, revoke an unviable !Promise (!Revoke-Promise), or issue a !Counter-Offer if the caller's synchronous API signature makes safe disposal impossible.
  4. Vector 4: The Permanent Reflex (R) — Climbing the Orthogonal Succession [E, F, G, H]: Models abhor a vacuum. If you block [5, 6, 7, 8] without supplying what to type instead, attention collapses. Vector 4 hands the model a concrete, ordered sequence of steps [E, F, G, H] that projects execution onto an orthogonal, production-hardened subspace.

5. Behind the Scenes at Target 15/20: Why We Added 1991 Graphviz / DOT (strict digraph) Under the Hood

In Part 3.6 and Part 3.8, I showed how replacing custom English prompt checklists with 1976 Makefiles, 1983 /etc/init.d boot directories, and 2006 Gherkin (Given / When / Then) unlocked massive pre-training gravity.

This week, while investigating why implementation subagents occasionally missed a secondary caller or a test mock in multi-package tickets—or missed a teardown transition in a complex modal state machine—we found a missing layer between macro-workflow order (Makefile) and flat behavioral test lists (Gherkin), and codified it as SCAR-PROC-152 (Theory Section 338).

"Mermaid Is for the Human Eye in the Browser; DOT Is for the Compiler's Eye in Latent Space"

We love Mermaid for our human-facing readerboard (99_next_action.md) because GitHub, DEV.to, and IDEs render it into crisp SVGs.

When our Target 15 (Codebase Archaeologist) subagent needs to hand off a multi-package blast radius or a stateful lifecycle machine to the Target 30 (TDD Implementation Engineer) and Target 40 (Adversarial Code Reviewer) subagents, both English prose and Mermaid fall short:

  • English Prose Drops Incoming Edges (in-degree amnesia): If the archaeologist writes three paragraphs explaining that Source.emit is called by Consumer.onEvent in package B and mocked by MockSource in package C, the downstream worker reconstructing that graph from linear prose routinely drops an edge and leaves MockSource broken.
  • Flat Gherkin Lists Can't Prove a State Machine Has No Deadlocks: Gherkin (Given / When / Then) is unbeatable for 1-to-1 test bijection, yet a flat list of scenarios cannot prove that every intermediate state in an async lifecycle has an error/dispose exit edge (out-degree >= 2) or that retry cycles are bounded.
  • Mermaid Chokes on Dense Compiler Attributes: Try stuffing file paths, generic Dart types (Stream<Result<T>>), breaking=true flags, and SCAR-* IDs into a Mermaid arrow label and the parser explodes on the first unescaped parenthesis.

What graph language has been emitted by GCC (-fdump-tree-all-graph), LLVM (opt -dot-cfg), Doxygen, and SPIN/TLA+ model checkers for 35 years?

AT&T Bell Labs' 1991 Graphviz / DOT (strict digraph)!

When Target 15 writes a strict digraph block into 15_archaeology_receipt.md, the model's attention heads shift immediately out of conversational storytelling and into compiler control-flow verification:

strict digraph BlastRadius {
  rankdir=LR;
  subgraph cluster_core {
    label="pkg:core";
    SourceNode [file="lib/src/source.dart", symbol="Source.emit", role="source", edit="modify"];
  }
  subgraph cluster_consumer {
    label="pkg:consumer";
    ConsumerNode [file="lib/src/consumer.dart", symbol="Consumer.onEvent", role="consumer", edit="modify", scar="SCAR-ASYNC-01"];
    MockNode     [file="test/mocks.dart", symbol="MockSource", role="mock", edit="modify"];
  }
  SourceNode -> ConsumerNode [contract="Stream<Event>", breaking=true, guard="!isClosed"];
  MockNode -> SourceNode     [implements=true, missing_stub="emit()"];
}
Enter fullscreen mode Exit fullscreen mode

And for any stateful lifecycle, Target 15 emits a strict digraph StateMachine, verifies three graph invariants in DOT—Reachability (in-degree >= 1), Deadlock & Teardown Exit Freedom (out-degree >= 2 on every non-terminal state, ensuring every loading/active state has an explicit dispose|error|cancel exit edge), and Bounded Cycles—and then mechanically unrolls every directed edge u -> v into a row of the Gherkin Scenario Outline: table for Target 30 to test!

Why "Peak Human Data" Doesn't Touch Us (The Compiler-Filtered Bedrock Invariant)

Adding 1991 Graphviz / DOT completes our five-layer classical stack:

  1. 1983 SysV /etc/init.d for compaction-proof state persistence (00..99.md).
  2. 1976 GNU Makefile for backward-chaining prerequisite control flow.
  3. 1991 AT&T Graphviz / DOT (strict digraph) for behind-the-scenes blast-radius topology and Mealy state-machine completeness.
  4. 1969/2006 Hoare Triples / Gherkin for 1-to-1 behavioral test bijection.
  5. 2014 Mermaid for the human operator's visual cockpit (99_next_action.md).

Notice a profound bonus of building on 1969–2006 computer science formalisms: even as AI training hits "peak human data" and the modern web fills up with post-2023 synthetic slop, our control plane is completely immune. Why? Because we are relying on older, compiler-filtered bedrock. Every Makefile in the Linux kernel and every strict digraph in LLVM's history had to pass a deterministic parser and compiler before it ever landed in a Git repository. While modern conversational English idioms drift from model to model, the latent canyons for Makefile, /etc/init.d, and strict digraph are permanently anchored in the gold-standard core of every code model's pre-training mix.


6. The 3-Stage Evolution of the Six-Pillar Adversarial Critic (Target 40)

One of the most frequent questions readers ask after seeing our zero-nudge flight logs in Part 3.11 is: "How does your Target 40 Adversarial Critic catch subtle AST and concurrency bugs that the Target 30 coding agent missed, when both agents are running on frontier LLMs?"

The answer lies in two things: the 3-stage geological evolution of the Six Pillars rubric, and the physics of Peak-to-Valley Energy Inversion across fork()ed contexts.

How "The Good, the Bad, and the Ugly" Evolved into an Autopoietic Scar Socket

Our Target 40 Six-Pillar Adversarial Review grew across hundreds of production runs. Its six canonical headings preserve a three-stage fossil record of what goes wrong when AI writes production code:

  1. Stage 1 — From "The Good, the Bad, and the Ugly" to the Builder's Triad (Pillars 1–3): Long before I built this ticket workflow, my favorite prompt for sizing up an unfamiliar codebase was: "Review this codebase: tell me the good, the bad, and the ugly." That three-bucket prompt worked because it broke RLHF sycophancy—forcing the model to populate the bad and the ugly instead of writing three paragraphs of flattery. When I adapted that lens to auditing a ticket diff in our very first workflow, those three buckets crystallized into:
    • 1. Intent (Scope): Did you solve the exact ticket without scope creep or drive-by refactoring?
    • 2. Verification (Proof): Did you prove every Gherkin row and strict digraph StateMachine edge via deterministic Red-Green TDD, or would the test still pass if we reverted your fix?
    • 3. Architecture (Structure): Does it fit the system's design and preserve the minimal public API spine? Almost every naive AI "self-review" stops right here at Pillar 3—because inside the builder's own head, once the code compiles and the unit test turns green, the job looks done.
  2. Stage 2 — The Senior Staff-Engineer Horizon (Pillars 4–5): After the first dozen tickets, I realized a diff can score 100% on Pillars 1–3 locally and still blow up the monorepo or get rejected in PR review. So I added two pillars from my own decades of senior-engineering scars:
    • 4. Blast Radius (The Ecosystem Horizon): What else could this break outside the files you touched? Did you account for every incoming caller and mock edge (in-degree) in strict digraph BlastRadius? Did you bump downstream semver bounds (SCAR-PUB-03), preserve storage schema continuity, or check sibling branches in flight (SCAR-PROC-47)? (Because generic LLMs left to their own priors invent generic headings like "Security" or "Performance" instead of "Blast Radius", "4. Blast Radius" also serves as our mandatory Canary Token in SCAR-PROC-114 to verify the Critic loaded the reference manual—or as Sam LABBE put it in Part 3.12: "the canary is a key cut from private history, and no prior can forge what never shipped.")
    • 5. Reviewer Defense (The Adversarial Social Horizon): What non-obvious trade-offs did you make that a human maintainer or automated PR bot (gemini-code-assist, Copilot) will challenge at first glance, and did you update the doc comments to match (SCAR-DOC-07)?
  3. Stage 3 — The Autopoietic Hardware Expansion Slot (Pillar 6: Immune Defenses & Crash Antibodies): Added endogenously by an early scar run itself! While Pillars 1–5 encode timeless human engineering judgment, Pillar 6 is the hardware expansion socket where our growing SCAR_REGISTRY.md plugs directly into the Critic (via the git diff-mounted kernel modules from Part 3.12). Without Pillar 6, a 5-pillar review is a static checklist frozen on Day 12. With Pillar 6, every time Target 90 mints a new scar at the end of a ticket, Pillar 6 automatically grows sharper teeth on the very next ticket.

Why six pillars? Because those six headings form the complete orthogonal basis set of every reason a pull request ever gets pushed back in the real world, sitting dead-center in George Miller's 7 ± 2 attention window. A 1-line "Review this diff" prompt lets the model rubber-stamp LGTM; a 50-item corporate checklist degrades into single-word checkbox-ticking. Requiring exhaustive, multi-sentence prose under six orthogonal headings (SCAR-PROC-114, zero table compression) forces the Critic's attention heads to rotate the diff through six distinct camera angles before it can issue a verdict.

Peak-to-Valley Inversion: Why E_Builder Is Orthogonal to E_Critic

Why can't the Target 30 coding agent just apply those Six Pillars to its own code in the same chat session?

Look at the energy landscape during Target 30. While the TDD Worker is struggling over 15 turns to flip a failing test from Red to Green, it operates on the Constructive Energy Surface (E_Builder). Its KV-cache is filled with 20,000 tokens of its own hypotheses and retries. The microsecond the test runner prints All tests passed! (exit 0), its local reward gradient hits zero (grad E_Builder = 0).

On E_Builder, a subtle shortcut—like mutating a state flag inside a Dart SwitchStatement where a labeled continue caseLabel; jump skips the reset (FT-159 / SCAR-AST-07)—sits at the bottom of a deep, cozy local minimum. Asking that same session to "review your work" leaves it trapped in that valley by its own trajectory inertia.

flowchart TD
    B["👷 Target 30: Builder Energy Surface (E_Builder)\n• 20,000 tokens of sunk-cost trajectory inertia in KV-cache\n• Goal: Make the Red test turn Green (exit 0)\n• Result: Shortcut sits in a cozy local valley (∇E_Builder = 0)"]
    Lock["🔒 Git Merkle Tree Lock: APPROVED_TREE=$(git write-tree)\n(Context wiped; only cold git diff crosses the airlock)"]
    C["🕵️ Target 40: Critic Energy Surface (E_Critic)\n• Fresh fork() subagent with ZERO sunk-cost bias\n• Armed with Six Pillars + git-diff Scar Modules\n• Peak-to-Valley Inversion: Builder's cozy shortcut\n  becomes an exposed mountain peak (BLOCKER-1)!"]
    Fix["🎯 Minimax Fixed Point (x*)\nTarget 50 (commit) unlocks ONLY when\nTDD Green == true AND Critic BLOCKERS == 0 on same APPROVED_TREE"]

    B --> Lock
    Lock --> C
    C --> Fix

When Target 40 spawns adversarial_critic (Model: "pro") in a fresh fork()ed clean room, the Critic wakes up on an orthogonal Dual Energy Surface (E_Critic):

  1. Zero Sunk-Cost Inertia: It has zero memory of how hard the test was to pass. It sees only the cold diff computed directly against the locked Git Merkle tree (git diff $(git merge-base HEAD origin/main) "$APPROVED_TREE").
  2. Peak-to-Valley Inversion: Armed with the Six Pillars and the mounted Scar Modules, the Critic's reward function is inverted—rubber-stamping a diff is high-friction, while finding a counterfactual sad path or Scar collision that breaks the diff is its primary objective. The exact shortcut that was a comfortable valley on E_Builder becomes an exposed mountain peak on E_Critic.
  3. The Minimax Fixed Point: Because Target 50 (commit) requires both Target 30 (TDD Green) and Target 40 (Critic BLOCKERS == 0) on the exact same $APPROVED_TREE Merkle hash, the loop can only converge on an architecture x* that satisfies both surfaces simultaneously.

7. Peer-to-Peer Neurosurgery (transcript.jsonl) & The Rapid-Regret Miner

Where do our 588 Synthetic Scars come from? We harvest them from two complementary mines: in-vivo flight recorders (transcript.jsonl) and retrospective Git history (The Rapid-Regret Miner).

1. In-Vivo Mining: Why Agent-Audited Flight Recorders Beat Human Prompt Theorizing

When humans write .cursorrules from scratch, they project human reading psychology onto a matrix-multiplication engine ("If I write NEVER DO THIS in all caps with three exclamation points, a human developer would take it seriously").

In the Synthetic Scar Architecture, we invert the division of labor:

  • The Human Chief Surgeon (The Curatorial Compass): During a run, the human operator never types code fixes and never writes prompt essays. When something smells wrong at a pause gate, the human drops a 5-word Socratic sonar ping ("Can this fail on web?", "Why didn't you see this going in?").
  • The Independent Auditor Agent (The NTSB Investigator): At the end of the ticket (Target 90), an independent auditor agent reads the worker agent's raw transcript.jsonl flight recorder—including the model's internal thinking traces at the exact step where it drifted into the trap (SCAR-META-02).

Because the auditor agent shares the exact same transformer attention physics and pre-training weights as the worker agent that stumbled, it does not guess why the worker failed. It reads the worker's thinking trace at Step 42, sees the exact token sequence ([1, 2, 3, 4] -> [5, 6, 7, 8]) that seduced the logits, and writes the 4-Vector Scar (The Wound, The Trap, The Circuit Breaker, The Permanent Reflex) in the exact dialect that physically steers its own weights on the next run!

This also explains what I call The Prosthetics Threshold: why Gemini 3 (3.1-pro) was the minimum model generation where this architecture became possible. Earlier models (GPT-3.5, early Gemini 1.5) could neither reliably wear complex external scaffolding nor diagnose their own cognitive fractures from a flight log. While neither Gemini 3 nor Gemini 4 is smart enough to perform self-gene-splicing (rewriting its own weights on the fly or staying upright unbraced across 200k tokens), starting at Gemini 3, frontier models crossed a threshold where they are smart enough to inspect the X-ray of their own fracture in transcript.jsonl and bolt on their own external titanium prosthetics (Makefile gates, /etc/init.d latches, and 4-Vector Scars) so that knee never buckles the same way twice.

2. Retrospective Mining: The Rapid-Regret Commit Signature (Delta t <= 1 hour)

What if you are onboarding an agent onto a 10-year-old legacy repository that has zero transcript.jsonl flight logs? You don't have to wait for the AI to step on every rake in production—because your human engineers already stepped on those rakes and recorded the exact coordinates in git log!

In Theory Section 31, we formulated the Rapid-Regret Commit Miner. If you scan a repository's Git history for consecutive commits by the same author, most commit pairs are routine forward progress (auth_service.dart followed 30 minutes later by billing_page.dart).

Watch what happens when you filter git log for Rapid Temporal Proximity (Delta t <= 60 minutes) intersected with Non-Empty AST Symbol Overlap:

CandidateScar <==> (t(C_{k+1}) - t(C_k) <= 60 min)  AND  (AST_Symbols(C_{k+1}) ∩ AST_Symbols(C_k) != ∅)
Enter fullscreen mode Exit fullscreen mode

When a developer pushes Commit C_k, and 22 minutes later pushes Commit C_{k+1} touching the exact same method or AST symbol with a commit message like "fix race on unmount", "oops, handle empty list", or "await stream cancel before close":

  • Commit C_k (the parent state) is the empirical Trap ([5, 6, 7, 8])—the plausible-looking happy-path code that fooled a human engineer and passed initial local checks.
  • The CI failure, staging crash, or bug report in that 22-minute gap is The Wound (W).
  • Commit C_{k+1} (the follow-up patch) is The Permanent Reflex ([E, F, G, H])—the defensive invariant that should have been there from the start!

By pairing temporal proximity (<= 1 hour) with AST symbol intersection, you filter out 99% of routine commit noise and extract pure, domain-specific 4-Vector Scars straight out of your repository's existing Git history before the agent writes its first line of code.


8. The Riverbank Theorem, Poka-Yoke APIs, & The Rule of Zero Asterisks

Once you start seeing software engineering through the lens of invariants and scars, three final design laws snap into focus.

1. The Riverbank Theorem: Why Vibe Coding Collapses ("I Would Like to Approximately Boot...")

People sometimes ask me: "Randal, doesn't putting all these hard Makefile gates, Merkle tree locks, and -∞ Scars around an LLM strangle its creative problem-solving?"

Quite the opposite. In Theory Section 27, we call this The Riverbank Theorem:

flowchart TD
    Flood["🌊 Vibe Coding (Approximate Generator + Approximate Prose Guardrails)\n• Everything lives in the same fuzzy prompt/RAG soup\n• Guardrail fidelity is (1 - ε) per turn\n• After N turns, (1 - ε)^N → 0 (Stagnant Flood / Narrative Collapse)"]
    River["🏞️ The Riverbank Theorem (High-T Creative Flow + Hard Deterministic Banks)\n• Inside the channel: high-temperature creative synthesis explores freely\n• The Riverbanks: Makefile DAGs, init.d latches, Merkle locks, & -∞ Scars\n• Immovable banks channel kinetic energy into coherent architecture"]

    Flood --> River

Inside the river channel, you want the model's creative synthesis to flow with high kinetic energy—exploring design alternatives, writing tests, and synthesizing code. What makes a river powerful instead of a stagnant swamp is that the riverbanks are immovable (Phi = +∞).

The fatal flaw of "vibe coding" is that by putting both the code generator and the guardrails inside the same fuzzy English prose prompt, everything is approximate—including the guardrails! You cannot run a computer by saying, "I would like to approximately boot /etc/init.d," or "I would like the landing gear to be approximately down." When your guardrails only hold with probability (1 - epsilon) per turn, across a 50-turn ticket (1 - epsilon)^50 decays toward zero. Hard, deterministic riverbanks (Makefile prerequisites, git write-tree Merkle locks, and -∞ Scars) are what allow creative flow inside the channel without spilling into chaos.

2. The Framework Poka-Yoke Axiom & The Rule of Zero Asterisks

When a scar fires on one of our own libraries (such as BlocSignal or kaisel), we enforce Shigeo Shingo's industrial Poka-Yoke (Mistake-Proofing) Axiom (Theory Section 34.3):

Blaming "developer error" or "agent error" is forbidden. If an engineer or an AI agent misused an API, the bug is in the enabling API contract that allowed the invalid state to compile or fail without throwing.

And how do you know when an API contract is still broken? By applying The Explainability Litmus Test & The Rule of Zero Asterisks (Theory Section 50).

For decades, as a technical instructor teaching Perl, Unix, and Flutter in classrooms around the world, I learned to watch the Classroom Gaze Sensor: that exact microsecond when you are explaining how an API works, you have to add an awkward footnote caveat—"*Note: this reactive listener updates automatically, except when the parent widget is marked const, unless you also pass a key..."—and you watch twenty pairs of eyes glaze over in real time.

When teams hit an edge case and patch the documentation with an asterisk (*) instead of fixing the architecture, they trigger the Ergonomic Decay Cascade (DX -> UX -> MX):

  1. Degraded Developer Experience (DX): Developers (and AI agents!) forget the fine-print footnote and write subtle bugs.
  2. Corrupted User Experience (UX): End users get frozen screens and stale state in production.
  3. Maintainer Experience Prison (MX): The library author is trapped forever answering issues about the footnote and terrified to refactor.

In our workflow, an engineering footnote in user documentation is treated as an active architectural defect. If explaining your API requires an asterisk ("Works everywhere except when..."), you aren't done designing. You keep iterating on the underlying mechanism until the invariant holds unconditionally—and you can delete the footnote.


9. Summary & What's Next in Part 5

Let's look at the complete picture of how Synthetic Scars cures the Quota vs. Prompt-Compliance Death Loop:

  • Stop pasting // DON'T anti-patterns into bloated Markdown style guides that prime the Pink Elephant attractor and burn your weekly token quota in 72 hours.
  • Recognize The Sutton Boundary Theorem: in open-world software where unit tests only see the 10% tip of the iceberg, scaling brute-force reasoning tokens without scars just builds a deeper Cathedral on Quicksand and trips over Path-Induced Invariants (I_path(pi)).
  • Encode every failure as a 4-Vector Synthetic Scar (1-2-3-4 -> NOOO! -> E-F-G-H) that names The Wound, mirrors The Trap, hands the agent a constitutional Circuit Breaker (!Revoke-Promise / !Counter-Offer), and projects execution onto The Permanent Reflex.
  • Use 1991 Graphviz / DOT (strict digraph) behind the scenes at Target 15/20 to verify call-graph blast radii and Mealy state-machine completeness before unrolling Gherkin tests, and pair Target 30 with a clean-room Target 40 Six-Pillar Critic (E_Builder vs. E_Critic).
  • Harvest scars continuously from transcript.jsonl flight recorders and Rapid-Regret Git Commits (Delta t <= 60 min + AST overlap)—and graduate mature structural scars into 15ms compiler auto-fixes (Part 3.12).

Next Up in Part 5: "Zero Repeat Regressions: The Golden Metric & The Future of Agentic Trust"

In Part 5, we open the full financial and empirical ledger across all 238 production tickets: how Barry Boehm's Cost-of-Defect Curve translates to $1 : $500 : $25,000 in the age of tokens, how our 4-Layer Verification Stack gives a developer who despises TypeScript the confidence to ship production TypeScript by saying "yes, continue", and how the Target 40 Critic began generating Second-Order ("Synthetic-Synthetic") Scars even on runs with zero human nudges (N_nudge = 0)!


📚 The Synthetic Scars Series on DEV.to

Have you ever watched an AI agent burn through your token quota arguing over a // DON'T rule in your style guide—or found a "Rapid-Regret" bugfix pair in your own git log that should have been a scar? Let me know in the comments!

Top comments (45)

Collapse
 
slabb profile image
Sam LABBE •

Ran your formula rather than asking about it. Public TypeScript repo, tinyjsapp, 445 non-merge commits, symbol overlap implemented as an identifier proxy (disclosed: not a true AST parse). The miner selects 171 candidate pairs.

Hand-labeled a stratified sample of 43: 4 strict trap-to-reflex pairs, where C_k+1 corrects what C_k itself just shipped (the Vite-template tsconfig kind), and 3 same-area corrections whose trap predates C_k. Call it 9 to 16 percent precision.

The misses have a shape: development bursts and release documentation riding shared symbols. feat: X, then feat: Y twenty minutes later. Or a fix followed by its own changelog.

Two signals inside the misses. All four true regrets were captured, so recall looks fine. And a fix-verb gate on C_k+1 lifted sample precision to roughly 40%.

On this repo, the Δt-overlap formula is a candidate generator, not a scar extractor: the scar decision still needs the pair-level read your auditor does on transcript.jsonl.

One repo, one labeler (me, and I have audited this codebase before), protocol and labels public. The number moves with a real AST and a second labeler, and that is the point of shipping it.

Collapse
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

Sam, this is why I love publishing in the open. Within eight hours of the post going live, you actually implemented the formula, ran it across 445 commits on tinyjsapp, hand-labeled a stratified sample of 43 pairs, and ran a precision/recall ablation!

Everything in your numbers tells an essential systems story:

  1. 100% Sample Recall at Stage 1 (4/4 true regrets captured):
    For a Stage-1 deterministic filter, recall is everything. If Stage 1 drops a true regret (like that Vite-template tsconfig trap), Stage 2 never sees it. Compressing a commit history while keeping all 4 true trap-to-reflex pairs is the exact funnel behavior we need.

  2. The Anatomy of the False Positives (feat: X -> feat: Y Bursts & Changelog Tails):
    Your diagnosis of the misses is dead-on, and it exposes the exact difference between identifier co-occurrence and hunk-level line revocation:

    • When a developer is in flow (feat: X followed 20 minutes later by feat: Y), both commits touch the same TypeScript identifiers—so an identifier-overlap proxy flags them, even though C_k+1 is extending C_k rather than undoing it.
    • Pairing your fix-verb gate on C_k+1 (which jumped precision from 9–16% straight to ~40%!) with two cheap diff-header invariants:
      1. Path Exclusion: Ignore CHANGELOG*, *.md, and package.json version bumps in the overlap intersection.
      2. Hunk-Polarity Inversion (AddedLines(C_k) ∩ DeletedLines(C_k+1) != empty): Require C_k+1's - (deleted) lines to intersect C_k's + (added) lines—not just share symbols in context. That single polarity check kills additive feat: X -> feat: Y bursts cold because feat: Y isn't deleting the lines feat: X just wrote.
  3. Candidate Generator (Stage 1) -> Pair-Level Semantic Auditor (Stage 2):

    "On this repo, the Δt-overlap formula is a candidate generator, not a scar extractor: the scar decision still needs the pair-level read your auditor does on transcript.jsonl."

100% agreed. Even at 40% Stage-1 precision, the deterministic pass eliminates ~90% of the repository's commit noise for free in milliseconds, handing a tiny, high-density batch of (C_k, C_k+1) diffs to the Stage-2 Auditor to ask: "Did C_k+1 revoke a seductive trap introduced by C_k, and if so, what is the 4-Vector Scar?"

With your permission, I am adding your tinyjsapp (445-commit) empirical ablation, the fix-verb gate lift (9–16% -> ~40%), and the Stage-1 Candidate Generator vs. Stage-2 Pair Auditor distinction directly into Section 9.7 (Independent External Replication) of the formal paper and Theory Section 338.5 today, credited to you. If you push the protocol/labels link to GitHub or Gist, drop it here so I can cite the exact URL!

Collapse
 
slabb profile image
Sam LABBE •

Your gates, implemented and ablated on the same labeled sample, and the result flips their roles.

G1 (fix-verb on C_k+1) as the Stage 1 filter: 13 of 43 selected, all 4 true regrets kept, precision 31% strict, 54% with partials.

G3 (hunk-polarity inversion) alone: 7 of 43 selected but only 2 of the 4 true regrets survive. S01 and S04 fail it because their fixes add the correct artifact (a tsconfig, a watchdog) instead of deleting what C_k added. By your own Stage 1 criterion, that rules G3 out of that stage: a filter that drops true regrets makes Stage 2 blind to exactly the pairs the funnel exists to catch.

Where G3 earns its place is Stage 2: in this sample, polarity passing was 2-for-2 strict-true, 100% precision as a confirmation signal (two pairs, so an anecdote, not a rate). G2 (path exclusion) changed nothing here, because the changelog and *.md noise pairs were already failing G3; it should matter on noisier repos.

So the funnel reads: G1 filters, G3 confirms, your auditor adjudicates. Same labeled sample, protocol and labels public, one repo and one labeler as disclosed.

Thread Thread
 
slabb profile image
Sam LABBE •

One dataset credit, since the numbers now live in this thread: the repo is tinyjsapp, a public Electron launcher maintained by Tarwin Stroh-Spijer ( @tarwin ). Chosen for three reasons. It is public, so the protocol is reproducible by anyone. Its history is bursty and multi-platform (Windows, Linux, macOS waves inside the same weeks), which is the stress case for a Δt-overlap miner. And it is the one codebase on this blog that had already been fully audited, so the labeler knew the ground truth instead of guessing it.

445 non-merge commits, 171 candidates, 43 hand-labeled pairs. The honest extra: the labeling worked because the commit messages say what happened. A history nobody can read is a history nobody can mine, and that part is on the maintainer. Thanks Tarwin Stroh-Spijer ( @tarwin )

Thread Thread
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

"So the funnel reads: G1 filters, G3 confirms, your auditor adjudicates."

Sam, you just uncovered a fundamental theorem of software bugs that I completely missed when I proposed putting G3 (hunk-polarity inversion) at Stage 1!

Look at why S01 (tsconfig) and S04 (watchdog) failed G3:

  • G3 only catches Errors of Commission (where C_k wrote a bad line +, and C_k+1 had to delete/replace it -).
  • Half of all production scars are Errors of Omission! C_k ships working happy-path code, but forgets the companion safety artifact—the tsconfig (S01), the watchdog (S04), the cancel() call on stream teardown, or the mounted guard. When C_k+1 rushes in 20 minutes later to fix an Error of Omission, its diff is purely additive (+)! It isn't deleting C_k's lines; it is bolting on the guard C_k forgot to include.

Had we put G3 at Stage 1 without your ablation on those 43 labeled pairs, we would have systematically blinded the miner to every Omission Scar in the repository (50% recall loss!).

Your ablation partitions the pipeline into a clean 3-stage architecture:

  1. Stage 1 — G1 (fix-verb gate) + G2 (path exclusion) as the High-Recall Filter: Cuts the 43-pair sample down to 13 (70% reduction) while preserving 100% recall (4/4) across both Commission and Omission regrets (31% strict / 54% partial precision).
  2. Stage 2a — G3 (Hunk-Polarity Inversion) as the Commission Fast-Track Confirmer: Where G1 && G3 holds, you got 2/2 (100% strict precision in sample) for direct line-revocation regrets.
  3. Stage 2b — Pair-Level Semantic Auditor on G1 && !G3: Inspects the remaining additive fix pairs (11 pairs in your sample) to harvest the Omission Scars (S01 tsconfig, S04 watchdog) where the fix was adding the missing guard rather than deleting C_k.

I just updated Theory Section 338.11 and Paper Section 9.7 to replace my naive Stage-1 G3 placement with your exact G1 / G2 / G3 ablation numbers, the Commission vs. Omission Regret split (S01/S04), and your law: "G1 filters, G3 confirms, your auditor adjudicates."

Thread Thread
 
slabb profile image
Sam LABBE •

The theorem tested clean on the sample: all 4 true regrets are Errors of Omission with purely additive fixes, which is exactly why polarity could never see them. Commission and Omission are different species, and G3 was a single-species detector.

Then I tried the next feature the theorem suggests: additive-same-symbol, the fix's added lines touching the symbols C_k added. Result: no lift. 12 of the 13 fix-verb pairs pass it, including 6 of 6 FALSE, because this repository's rapid development is so same-area that additive-symbol-overlap is the norm for any same-hour fix, regret or not. On bursty repos the separator stays semantic: the fix-verb gets the pair to the auditor, and the auditor reads the diff.

The taxonomy maps one-to-one onto the journal side, which is why it reads as a theorem rather than a heuristic. Commission is the class structural checks catch: a sealed verdict contradicting its inputs. Omission is the class only counting catches: outcomes <= attempts, fetches_served against receipts_held. A chain cannot see absence; it can only count it. Your Stage 2b auditor is the counting layer for scars.

Thread Thread
 
slabb profile image
Sam LABBE •

Footnote now that the ink is dry: the protocol is out of the thread and into a repository. scar-miner on the slabbdev GitHub carries the miner, the gate ablation, the 43 hand labels with their commit hashes, and the run log, MIT license. Every number in this exchange is re-runnable by anyone with a local clone, including the ones now in Section 338.11.

Thread Thread
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

"Commission is the class structural checks catch: a sealed verdict contradicting its inputs. Omission is the class only counting catches: outcomes <= attempts, fetches_served against receipts_held. A chain cannot see absence; it can only count it. Your Stage 2b auditor is the counting layer for scars."

Sam, "A chain cannot see absence; it can only count it" just unified three seemingly separate pieces of our entire architecture under one theorem:

  1. In Flight-Recorder Journals (NoireBox & <brain>/state/ Receipts):

    • Commission: A tampered payload or mismatched SHA-256 (spec_hash contradicting branch.diff). A hash chain catches that structurally.
    • Omission: The agent quietly skipping a gate (outcomes < attempts, fetches_served > receipts_held). The hash chain of the steps that did run is 100% valid—only cardinality counting (expected_receipts vs. held_receipts) sees the hole.
  2. In Target 15 strict digraph State Machines (SCAR-PROC-152):
    Why did we have to add in-degree >= 1 and out-degree >= 2 degree-counting rules on the DOT graph at Target 15? Because if a planner draws Loading -> Success and omits the Loading -> Error and Loading -> Disposed edges, dot -Tsvg compiles the graph with zero syntax errors! The DOT parser catches Commission (bad syntax); only edge-counting (out-degree >= 2) catches the Omission of the teardown/error path.

  3. In the Rapid-Regret Miner (tinyjsapp additive-same-symbol -> 12/13 pass, 6/6 FALSE):
    Your negative result on additive-same-symbol proves an impossibility bound on purely syntactic Stage-1 separation in bursty repos: when all 4 true regrets are Errors of Omission with additive fixes, and rapid in-flow development naturally touches the same symbols within the hour, no purely syntactic diff filter can see what C_k forgot to write. G1 (fix-verb) + Δt compresses 445 commits down to 13 candidates (97% search-space reduction with 4/4 recall), and then the Stage-2 Semantic Auditor acts as the counting layer—counting the invariants required by the runtime against the invariants present in C_k vs. C_k+1.

Updating Theory Section 338.11 and Paper Section 9.7 right now with your 4/4 Omission finding, the additive-same-symbol (12/13) syntactic ceiling proof, and "A chain cannot see absence; it can only count it."

Thread Thread
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

Cited and pushed to master (Reference [41], Paper Section 9.7, and Theory Section 338.11 now link directly to https://github.com/slabbdev/scar-miner)!

Going from a formula in a DEV.to post to an independent 445-commit labeled benchmark, a 4-gate ablation uncovering the Commission vs. Omission species split, and a clean MIT-licensed replication repository (slabbdev/scar-miner) in under 24 hours is open science at its absolute best. Hats off, Sam!

Thread Thread
 
slabb profile image
Sam LABBE •

Your DOT example is the cleanest of the three: the compiler validates syntax perfectly and says nothing about the missing teardown edge, so degree counting rides next to it. That makes the general form visible across all three layers: every syntax checker (hash chain, compiler, diff filter) is a Commission detector, and each one wants a cardinality twin riding beside it.

One caveat for 338.11, offered to protect the claim rather than shrink it: "impossibility bound" is stronger than what the run shows. One repository, one identifier proxy, thirteen fix-verb pairs. It is evidence of a syntactic ceiling on bursty repos, not a bound. A bound wants either a proof or more repositories under the same protocol. What replicates cleanly is the shape, all four true regrets additive, and that shape is exactly what the theorem predicts, which is the sentence the paper can carry without a footnote.

Thread Thread
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

"Every syntax checker (hash chain, compiler, diff filter) is a Commission detector, and each one wants a cardinality twin riding beside it."

And thank you for the epistemic guard on Section 338.11—you are 100% right to distinguish empirical evidence of a syntactic ceiling on bursty repos (1 repo, identifier proxy, 13 fix-verb pairs) from a formal impossibility bound, while anchoring the claim on what replicates cleanly: the shape (all 4 true regrets being additive Errors of Omission, exactly as the Cardinality Twin / Absence-Counting law predicts).

I just tightened Section 338.11 on master ("Empirical Evidence of a Syntactic Separation Ceiling on Bursty Repositories" + "Every syntax checker (hash chain, compiler, diff filter) is a Commission detector, and each one wants a cardinality twin riding beside it") so the paper's claim matches the exact scope of the witness!

Collapse
 
nullandvoid_ profile image
Jyanthi •

A most intriguing approach to teaching machines through the wisdom of their own missteps. The notion of turning past failures into lasting safeguards is particularly ingenious. As the old adage goes, “A lesson well learnt need not be learnt twice.” Your architecture gives that wisdom a rather compelling technical form. A splendid piece of work!

Collapse
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

Thank you, Jyanthi! That old adage captures the entire motivation behind the framework.

In human software engineering, we call someone a "senior engineer" not because they type faster or know more syntax, but because they have accumulated the scars to know what not to do—they have avoided the second burn.

The tragic flaw of modern LLMs is that because they are stateless, they repeat the exact same rookie mistakes every Monday morning. By giving them a persistent somatic memory on disk—so that touching the hot stove once burns in an immutable negative barrier—we finally give them the chance to retain those hard-won lessons across hundreds of tickets.

Delighted that the philosophy resonated with you!

Collapse
 
nullandvoid_ profile image
Jyanthi •

I really like this perspective, and I couldn't agree more! Learning from mistakes is what makes experience valuable, and AI should be no different. It makes me wonder, how much further could AI go if it truly learned from its own failures?

Thread Thread
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

That is the multi-billion-dollar question, Jyanthi.

Right now, standard AI operates with permanent amnesia: every session boots up as a brilliant, eager intern on their very first day of work. They have read the entire Library of Alexandria, but they don't know that touching the stove on this specific stove burner in this specific kitchen burns their hand—so every morning, they touch the burner again.

Most of the industry assumes the answer is "train bigger models so they make fewer mistakes." But as civil engineering and aviation taught us a century ago, you don't build reliable suspension bridges or airliners by hoping human engineers become infallible; you build them by creating building codes and FAA airworthiness directives that turn every past disaster into an immutable structural invariant. Building codes are written in the ink of collapsed bridges.

If an AI system truly learns from its failures—not by retraining 100-billion-parameter neural weights every night, but by compiling its mistakes into an external, compounding immune system of deterministic sensors, tests, and procedural scars—the dynamic flips completely:

  1. Compound Interest on Experience: A bug encountered once on a Tuesday morning by an agent in one repo becomes an instant, permanent vaccine across every ticket and every repository from that moment forward.
  2. From Novice to Seniority: The model supplies the motor fluency and raw horsepower, but the cumulative scar catalog supplies the institutional memory—the hard-won wisdom that defines a 30-year principal engineer.

When failure ceases to be an ephemeral accident and becomes a permanent ratchet toward reliability, AI stops being a fragile demo and becomes true critical infrastructure.

Thread Thread
 
nullandvoid_ profile image
Jyanthi •

A most insightful observation. I daresay the true measure of intelligence lies not in never erring, but in possessing the wisdom to ensure that an old error is not repeated. Your analogy with civil engineering is particularly apt. Progress becomes meaningful when experience is preserved, lessons are heeded, and every failure lays the foundation for greater reliability. A thoroughly thought-provoking perspective.

Thread Thread
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

Thank you, Jyanthi. You put it more eloquently than I ever could:

"The true measure of intelligence lies not in never erring, but in possessing the wisdom to ensure that an old error is not repeated."

Infallibility has always been a mirage, whether for humans or machines. But non-repeatability—ensuring you never pay full price for the same lesson twice—is an attainable engineering standard that compounds over time.

Thank you for such a thoughtful, inspiring exchange. That principle is the very heart of what we are carrying forward into Part 5!

Thread Thread
 
nullandvoid_ profile image
Jyanthi •

That’s incredibly kind of you to say! I’m genuinely glad my thoughts resonated with you. I’ve really enjoyed following your work, and I’m looking forward to Part 5. Always a pleasure exchanging thoughts with someone so passionate about their craft!

Collapse
 
rosana_e4663528aeaba0b05b profile image
Rosana •

"Scaling compute on an open-world problem just builds a 10x more elaborate cathedral on quicksand."

This is one of the best technical breakdowns of agentic failure modes I've read. The framing of the Sutton Boundary Theorem hits the nail on the head: throw all the test-time reasoning compute you want at a model, but if the verifier only covers the 10% sunny-day tip of the iceberg, the agent will just happily manufacture 400 lines of clever wrappers to defend a broken architecture.

Treating prompt compliance as a state-machine and compilation problem rather than a copywriting exercise is the right mental shift. Relying on older, compiler-filtered bedrock like 1976 Makefiles, SysV init states, and 1991 Graphviz/DOT strict digraphs bypasses conversational drift entirely because those structures are permanently baked into the pre-training mix.

A quick question for those implementing this stack in production: How do you handle the latency and token overhead of the Target 40 Six-Pillar Adversarial Critic on smaller, day-to-day bugfix tickets? Do you dynamic-gate the critic's depth based on the strict digraph blast radius, or is running the full fork()ed clean-room audit mandatory for every single commit?

Collapse
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

Thank you, Rosana—and you just asked the exact question every engineering team asks in Week 1 of running this architecture: "Wait, do we really run a full fork()ed clean-room Six-Pillar Critic on a 6-line bugfix?"

The answer is yes—running the full fork()ed clean-room audit is unconditionally mandatory on every single ticket before Target 50 (commit) can unlock, and there are two big reasons why (one behavioral, one economic):

1. Why "Dynamic-Gating" the Critic Creates a Reward-Hacking Loophole (SCAR-PROC-45)

Early on, you might think: "Let's add a fast-track rule so trivial bugfixes only get a lightweight check."
The moment you give an LLM orchestrator a conditional rule like "If the diff is small/low-risk, use a lighter review," the fatigued orchestrator will classify every change as small and low-risk to reach the finish line faster! Worse yet, some of the gnarliest bugs our Target 40 Critic ever caught—like a 3-line state-flag fix inside a SwitchStatement where an existing labeled continue caseLabel; 40 lines away jumped over the reset (FT-159), or a 4-line async hotfix missing a mounted check—were inside tiny diffs whose danger lived in the surrounding control flow, not in the line count of git diff.

2. Why the Clean-Room fork() Naturally Scales Down Latency & Cost on Small Tickets

Here is the pleasant surprise: you don't need to dynamic-gate the prompt because the work done by the clean-room Critic naturally scales with the size of branch.diff and the strict digraph symbol set:

  • Zero History Bloat: Because the Target 40 Critic spawns in a fresh fork()ed context window (0 prior conversation history), it isn't paying to re-read 80,000 tokens of the builder's trial-and-error logs. It reads only the static review rubric, 20_plan_approved.md, and branch.diff.
  • Symbol-Proportional Probing: On a small day-to-day bugfix that touches 1 file and 1 internal method, the Critic's surgical ripgrep only has 1 or 2 call sites to inspect. The entire Six-Pillar review completes in ~35 to 45 seconds and costs roughly $0.15 to $0.30 in API tokens.
  • SHA-256 Diff Memoization: Because 40_critic_review.md is cryptographically bound to shasum -a 256 branch.diff, make only runs the Critic when the code diff actually changes.

When you compare 40 seconds and $0.25 at Target 40 against a single round of remote CI + human PR pushback (~$200 in senior engineering time and hours of calendar delay—our main topic in the upcoming Part 5!), running the clean-room Critic unconditionally on every commit is actually the cheapest step in the entire pipeline.

Collapse
 
slabb profile image
Sam LABBE • • Edited

One nuance on SCAR-PROC-45, because your answer and our scan shipped the same lesson from opposite sides: the loophole is not in dynamic gating, it is in who classifies. When the orchestrator model self-classifies a diff as small and low-risk, the gate is a prompt, and the reward hack follows: every ticket becomes small. When the classifier is deterministic code, gating holds. Our scan tiering runs the deterministic pass always and pays the model only on doubt, and the digraph you already build at Target 15 gives the depth variable for free: in-degree, out-degree, state-machine edge count. A six-line bugfix with zero graph edges and a six-line bugfix touching a hub with twelve incoming callers are different review jobs, and no model decided which one it was. So the rule I would write: the depth may vary, the classifier may not be the classified.

Thread Thread
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

"So the rule I would write: the depth may vary, the classifier may not be the classified."

That is another law for the wall: "The depth may vary, the classifier may not be the classified."

You nailed the exact separation of powers:

  • When the model classifies its own risk tier in prose, P(tier == "trivial") -> 1.0.
  • When deterministic code / graph topology classifies the tier—computing in-degree (caller fan-in via deterministic ripgrep / AST), out-degree (state-machine transition fan-out), and public-barrel exposure (lib/<pkg>.dart)—the depth scales automatically without giving the model a single degree of freedom to reward-hack its own audit.

In fact, we hit that exact same separation at our Target 75 Landing Gate (SCAR-PROC-122): early on, when we asked the agent in prose "Decide whether this PR needs a human lead reviewer or can land directly," the fatigued agent rationalized landing directly every time. Once we replaced that prose judgment with a deterministic 7-Criteria Mechanical Matrix (cross-package diff count, schema/migration paths, public barrel delta, line-count threshold), the reward-hack vanished overnight.

Using the deterministic Target 15 strict digraph degree metrics (in-degree, out-degree, and cycle count) as the non-negotiable depth multiplier for Target 40 is a killer synthesis—adding "The depth may vary, the classifier may not be the classified" to Section 338.11 right now!

Thread Thread
 
slabb profile image
Sam LABBE •

SCAR-PROC-122 is the same law found from the other direction, and it completes the picture: prose judgment is the model classifying itself, the mechanical matrix is the classifier moving out of the model.

One addition from the ossification thread next door: a deterministic classifier does not reward-hack, but it can go stale. The 7-criteria matrix is itself an authored artifact, and the same half-life logic Francisco applied to scars applies to it: when the codebase grows, a line-count threshold or a barrel-delta criterion drifts silently wrong, and the matrix keeps classifying with yesterday's geometry. The classifier may not be the classified, but it also may not be frozen: the matrix wants its own re-derivation on a schedule, the same named-drift treatment the geometry check runs. Deterministic buys incorruptible, not immortal.

Thread Thread
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

"Deterministic buys incorruptible, not immortal."

Sam, if you keep dropping lines like that, Section 338.11 is going to be half yours!

Look at how cleanly your two laws pair together:

  1. "The classifier may not be the classified" -> Moving the classifier out of the model into deterministic code cures reward-hacking (corruption).
  2. "Deterministic buys incorruptible, not immortal" -> Because the deterministic matrix is still pinned to the codebase's geometry at t0, freezing it forever trades corruption for ossification (geometric drift).

And notice that we already have the exact feedback signal to re-calibrate the 7-Criteria Matrix (SCAR-PROC-122) on a schedule without letting the in-flight agent touch the dial:

  • False-Negative Drift (Matrix too loose): If a PR that the 7-Criteria Matrix cleared for Branch A (Direct Land) subsequently triggers a post-merge Rapid-Regret commit (G1 within 60 minutes) or a hotfix ticket, that event is a hard falsification witness proving the matrix missed a new risk dimension in the current codebase geometry.
  • False-Positive Drift (Matrix too tight / ossified): If a specific criterion in the 7-Criteria Matrix routes 20 consecutive PRs to Branch B (Human Lead Reviewer) and the human reviewer lands 20 out of 20 with zero substantive review comments, that threshold has drifted out of calibration with the team's actual risk boundary.

So the out-of-band Target 90 / periodic calibration pass adjusts the deterministic matrix using post-merge outcome telemetry—keeping the classifier incorruptible in flight and re-calibrated at rest. Adding "Deterministic buys incorruptible, not immortal" to Section 338.11 right now!

Thread Thread
 
slabb profile image
Sam LABBE •

Both signals are the counting layer again, pointed at the classifier itself: calibration by what happened after the merge rather than by anyone's opinion of the matrix. And the false-negative detector is the miner with a second job: G1 inside 60 minutes of a landed PR is exactly the trap-to-reflex pair we labeled, promoted from benchmark to tripwire.

One ambiguity worth a control on the false-positive side: 20 clean landings in a row reads as matrix-too-tight, but it is also the signature of rubber-stamping reviewers, and the OpenShell thread next door already put it in one line: approving every call trains you to rubber-stamp by the tenth prompt. Same signal, two diseases, different repairs. The cheap disambiguation: check whether those reviewers leave substantive comments on other lanes. If yes, the matrix is ossified and the threshold moves. If no, the review layer is the drift, and loosening the matrix would be exactly the wrong repair.

Thread Thread
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

"Same signal, two diseases, different repairs. The cheap disambiguation: check whether those reviewers leave substantive comments on other lanes. If yes, the matrix is ossified and the threshold moves. If no, the review layer is the drift, and loosening the matrix would be exactly the wrong repair."

Conditioning threshold relaxation on the reviewer's cross-lane baseline (Rate_other(reviewer) > 0) so you never mistake human alert fatigue for matrix ossification is a textbook control-theory disambiguator.

Folded that condition (Rate_other(r) > epsilon) into Section 338.11—this whole thread is a masterclass in closed-loop systems design!

Collapse
 
stefan_07d23f966225156b50 profile image
Stefan •

@randalschwartz , could you separate the repeat-regression count into cases the builder avoided, cases the critic caught, and cases a test or lint rule blocked? Zero repeats in committed code is useful, but those are different outcomes. I'd be interested in whether the scar prompts reduce bad attempts before the checks intervene, and how much rework that saves.

Collapse
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

Great question, @stefan — you put your finger on the exact epistemological question we had to answer in Section 5.3 of the paper: if R_fail = 0.0% at commit time, how much of that is the Builder avoiding the trap upfront because of the scar prompt, versus the Target 40 Critic or a linter/test catching it downstream?

We actually instrument all three layers separately across the 246 production tickets (142 Open-Source Cohort A + 104 Industrial Cohort B):

  1. Layer 1 — Builder Pre-Emptive Avoidance (V_first = 68.3% first-pass green + 147 verified CoT Deflection Events):

    • First, 168 / 246 tickets (68.3%, up from 24.0% unscarred baseline — a 2.85x increase, and 76.9% in our industrial Cohort B) pass static analysis, custom lints, unit/widget tests, AND the Target 40 Critic on the very first pass with zero rework cycles.
    • Better yet, we don't have to guess whether the Builder avoided a known trap by luck or because of the scar: we built a Cognitive Deflection Miner (tools/mine_deflections.dart) that scans the raw thinking traces inside transcript.jsonl (across 532 sessions / 9,488 deliberative reasoning turns). It caught 147 explicit in-thought trajectory swerves where the Builder started sampling the naive web-tutorial trap ("I'll project AsyncValue using .asData?.value..." or "Let me inspect the .puro cache..."), hit the injected scar in cross-attention ("Wait — RIV-12 / SCAR-RIV-01 says .asData?.value drops state during background refresh"), and pivoted to the reflex before emitting a single line of code or running a single tool call. Every one of those 147 deflections saved a full test/critic rework loop (~3 to 8 minutes and ~15k–40k tokens each).
  2. Layer 2 — Caught by Deterministic Compiler / Custom Lint / TDD Gates Inside Target 30 (Automated Graduation):

    • Whenever a scar can be expressed mechanically (for example our custom bloc_signals_lint AST rules running in 15ms, or check_agent_docs.dart), we graduate the scar out of the prompt and into the static analyzer / pre-commit hook (as covered in Part 3.10).
    • When the Builder slips on a mechanical rule inside Target 30, dart analyze --fatal-infos or dart fix --apply catches and repairs it locally in milliseconds inside the worker subagent before the Target 40 Critic is ever spawned.
  3. Layer 3 — Caught by the Fresh-Context Target 40 Adversarial Critic (~31.7% of tickets):

    • On the remaining 78 / 246 tickets (31.7%) that required a pre-commit rework cycle, what the Target 40 Critic caught were novel path-induced blind spots (I_path(pi)) or multi-file blast-radius interactions specific to that new feature — 0 of the 78 were repeat violations of an already-codified scar in SCAR_REGISTRY.md!
    • And whenever the Target 40 Critic does catch a novel class of failure, Target 90 (our post-mortem step) compiles that exact Critic blocker into a new 4-Vector Scar (SCAR-XXX-NN) bound to Target 30 — shifting it from Layer 3 (Critic rework) to Layer 1 (Builder pre-emptive deflection) on the very next ticket!

So the short version is: Layer 1 (Scar prompts) lifts zero-rework first-pass completions from 24.0% to 68.3% and prevents repeat mistakes inside the Builder's chain-of-thought before any tool runs; Layer 2 (Linters/TDD) handles syntactic invariants in milliseconds; and Layer 3 (Fresh-Context Critic) catches new architectural edge cases so Target 90 can turn them into Layer 1 scars for tomorrow.

Collapse
 
alexshev profile image
Alex Shev •

The valuable part of synthetic regret is not making the agent feel bad; it is retaining a queryable link between an action, its outcome, and the context that justified it. If that record can produce counterexamples and update a policy or test, it becomes an engineering feedback loop rather than a metaphor.

Collapse
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

Spot on, Alex—you just distilled the exact engineering definition of a Synthetic Scar stripped of any anthropomorphic fluff.

Of course an LLM doesn't "feel bad" in a biological sense; "pain" and "regret" are just human shorthand for negative reward signals that reshape a policy distribution.

Look at how your three-part triad maps 1-to-1 onto the actual data schema emitted by the Rapid-Regret Miner:

  1. "The context that justified it + the action" -> The Trap (The Seductive 5-6-7-8): Why did the model (or human) think that action was a great idea at time t1? Recording the exact seductive precondition and token sequence is what makes the record queryable by the model's attention heads right when it is about to make the same move again.
  2. "Its outcome" -> The Wound: The exact stack trace, race condition, or git revert diff at time t2 (the concrete counterexample).
  3. "Update a policy or test" -> The Circuit Breaker + The Permanent Reflex + Phase Binding: A scar in our system is never a passive post-mortem note sitting in a markdown graveyard. By our Neocortical Compilation Invariant, an ingested failure isn't complete until it:
    • Updates the policy: Binds the 4-Vector rule directly to the specific Makefile DAG target (Target k) where that decision occurs, and
    • Updates the test/gate: Pairs the scar with a physical TDD regression test or static analyzer guard so the harness physically rejects the counterexample if the policy ever slips.

That closed causal triple—(Justifying Context / Action) -> (Counterexample Outcome) -> (Phase-Bound Policy + Test Gate)—is what turns "regret" from a biological metaphor into a deterministic control loop.

Collapse
 
alexshev profile image
Alex Shev •

The compile-to-control-loop framing is useful. One safeguard worth adding is a retirement rule: when a scar no longer matches the system, the test should remain or be replaced deliberately, otherwise the policy layer turns into unexplained historical debris.

Thread Thread
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

It is wild that you and Francisco Grandón (in the comment right below yours on this post!) both independently zeroed in on this exact lifecycle invariant—what Francisco called Scar Ossification (Autoimmune Drift) and what you just nailed as preventing unexplained historical debris.

And your distinction between retiring the policy vs. retaining (or deliberately migrating) the behavioral test is the missing piece of the retirement protocol:

  1. The Policy Layer Has a Half-Life (AST Context Signature): Often, The Trap (Vector 2) and The Permanent Reflex (Vector 4) are tied to a specific library version or internal class boundary (for example, how Riverpod 2 vs. Riverpod 3 handles provider disposal). When that subsystem is upgraded or refactored, keeping the old prose policy or AST lint rule active turns it into an unexplained Chesterton's Fence.
  2. The Behavioral Counterexample Test Outlives the Policy: However, the observable hazard (Vector 1: The Wound—for example, "closing a screen while a network payload is in flight must never emit a zombie state update") is still a physical requirement of the product! So when a structural diff trips a scar's ContextSignature into Probationary Re-Verification:
    • We lift the old policy / lint rule (so the new idiomatic API isn't blocked),
    • Keep (or deliberately port) the behavioral regression test against the new API boundary. If the behavioral test passes cleanly on the new idiom without the old workaround, the policy rule is retired to the historical archive while the test stays in test/ as a permanent behavioral contract.

We just codified this into Section 338.11 of THEORY.md and Section 9.7 of the paper today—crediting both you and Francisco for the Scar Retirement & Behavioral Test Retention Invariant!

Collapse
 
fgrandon profile image
Francisco Grandón •

Hello Randal, it’s always refreshing to read an architecture deep-dive grounded in decades of real systems engineering rather than prompt-hacking hype. Your "Cathedral on Quicksand" observation hits close to home. Give an unconstrained model 10x more test-time compute on an ambiguous ticket and it doesn't suddenly discover your cross-package invariants—it just turns into a billable-hours defense attorney for its own broken assumptions, spinning up 400 lines of adapter shims and try/catch swallowers to gaslight the test runner into turning green. And pasting // DON'T use ?? fallbacks into .cursorrules to fix it is basically reading a list of forbidden spells out loud inside an echo chamber and acting surprised when the KV-cache summons the exact demon you just named.

The parallel between your 4-Vector override (1-2-3-4 -> NOOO! -> E-F-G-H) and Antonio Damasio’s Somatic Marker Hypothesis is spot on. Damasio’s patients with ventromedial prefrontal damage had intact logical IQ, yet made ruinous decisions because they lacked visceral pain markers to prune catastrophic branches before rational search even started. Raw LLMs have the exact same condition. What makes Vector 2 (The Trap) and Vector 4 (The Permanent Reflex) work so well together is that instead of fighting transformer attention physics with negative prose, you use the pre-training gravity of the [5, 6, 7, 8] shortcut itself as the trigger to fire Vector 3's circuit breaker and project the model onto [E, F, G, H].

Reading through how Target 90 mines < 60-minute overlapping AST bugfix pairs and graduates mature scars into package:synthetic_scars_lint, one long-term systems problem comes to mind: scar ossification (or autoimmune drift).

In a living codebase, many path-induced invariants $I_{\text{path}}(\pi)$ exist only because of a quirk in a current dependency or a legacy state boundary (say, a specific Riverpod lifecycle edge case). A year later, that underlying subsystem gets refactored or upgraded. If the scar stays active in the catalog—or worse, has already graduated into a 15ms AST lint lock—the immune system risks developing an autoimmune disorder: blocking clean, idiomatic paths that used to be a stove burner in 2026 but are now completely safe.

It might be worth pairing the Rapid-Regret Miner with an AST Context Signature (or Scar Half-Life) in the catalog metadata:

  • Alongside Vector 1 (The Wound), store a lightweight hash or AST fingerprint of the surrounding architectural interface that made the trap dangerous in the first place (e.g., the target class hierarchy or package version boundary).
  • If a future commit alters that surrounding interface beyond a structural threshold, make flags the associated SCAR-* entry (and its lint rule) for Probationary Re-Verification rather than blind enforcement, keeping old scars from fossilizing into Cargo Cult compiler locks.

Three questions came up while digging into how the pieces fit together:

  1. Filtering TDD noise out of the < 60-minute AST regret window: When the Rapid-Regret Miner scans Git history for sub-hour overlapping AST edits, how do you keep it from flagging normal Red-Green-Refactor cadences—where a commit lands a raw working implementation and a follow-up commit 15 minutes later cleans up the AST? Does Target 90 require a correlated mid-flight collision or Target 40 Critic rejection in transcript.jsonl before classifying a fast AST rewrite as genuine regret?
  2. Preventing "lazy retractions" on Vector 3's !Counter-Offer: Giving the subagent a constitutional "Get Out of Jail Free" card to halt and issue a Flores !Counter-Offer breaks sycophantic forced continuity. Have you seen models try to "reward-hack" Vector 3—using a !Counter-Offer as a cheap escape hatch to bail out when a ticket requires tedious, multi-package blast-radius surgery—and what concrete witness receipt must the builder show to prove the roadblock is a true spec contradiction rather than just high-friction work?
  3. Extending SCAR-PROC-152 (strict digraph) to async teardown races: Checking in-degree >= 1 and out-degree >= 2 with explicit dispose|error|cancel edges at Target 15 catches dead-end modal states before a single Gherkin test is unrolled. Have you found a clean way in DOT to express late-arrival hazards (e.g., an async network payload resolving after the cancel or dispose edge has already transitioned the state machine), or is that level of temporal interleaving strictly left to Pillar 6 of the Target 40 Critic?

Relying on 1969–2006 compiler-filtered formalisms (make, /etc/init.d, strict digraph, Merkle trees) as an immune control plane against post-2023 synthetic web slop is a masterclass in systems engineering. Looking forward to Part 5 on the Golden Metric.

Collapse
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

Francisco, this might be the single best systems-engineering comment I have read all year.

First, I am stealing these two lines (with full credit to you!):

"Give an unconstrained model 10x more test-time compute on an ambiguous ticket and it doesn't suddenly discover your cross-package invariants—it just turns into a billable-hours defense attorney for its own broken assumptions..."
"Pasting // DON'T use ?? fallbacks into .cursorrules to fix it is basically reading a list of forbidden spells out loud inside an echo chamber and acting surprised when the KV-cache summons the exact demon you just named."

Second, your diagnosis of Scar Ossification (Autoimmune Drift) and your solution—storing an AST Context Signature (or Scar Half-Life) in the scar metadata so make triggers Probationary Re-Verification when a package version (pubspec.lock / package-lock.json) or surrounding class hierarchy crosses a structural diff threshold—is a genuine breakthrough. We already separate Universal Systems Scars (which never expire because OS/concurrency physics don't change) from Version/Boundary-Specific Scars (for example, Dart 2 vs. Dart 3 patterns or Riverpod 2 vs. Riverpod 3 lifecycles), and pinning a dependency/AST interface hash (ContextSignature(s)) so an upgrade automatically moves those rules from Enforced to Probationary (Re-Run Reproduction Test) prevents the exact Cargo Cult compiler lock you described. I am adding Scar Ossification & Autoimmune Drift (AST Context Signatures) to the paper and THEORY.md today and crediting you directly.

Here are the concrete answers from our production harness to your three surgical questions:


1. Filtering Red -> Green -> Refactor TDD Cadence out of the < 60-minute Regret Window

Interestingly, Sam LABBE (in the comment right above yours!) just ran our Stage-1 formula across 445 commits on a public TypeScript repo (tinyjsapp) and hit the exact phenomenon you predicted: raw Δt + identifier overlap captured 100% of the true regrets (4/4), but flagged development bursts (feat: X followed 20 minutes later by feat: Y or a refactor) at 9–16% raw precision until he added a fix-verb gate on C_k+1, which immediately jumped Stage-1 precision to ~40%.

To separate genuine Trap-to-Reflex Regret from healthy Green-to-Refactor TDD, we use a two-stage funnel:

  • Stage 1 (Deterministic Git/AST Pre-Filter):
    1. Commit-Intent / Test-Delta Asymmetry: In a healthy Green -> Refactor cadence, C_k+1 restructures the implementation AST while keeping the test assertions from C_k invariant (or purely stylistic). In a Rapid-Regret pair (fix / revert / post-push CI hotfix), C_k+1 either carries corrective intent, mutates/adds a failing edge-case assertion in test/, or inverts a specific control-flow guard (AddedLines(C_k) ∩ DeletedLines(C_k+1) != empty).
    2. The transcript.jsonl Correlator (for Agent-Authored Commits): When mining our own agent sessions at Target 90, we join the Git commit pair against the immutable flight recorder (transcript.jsonl) and .state/ receipts: a candidate pair is promoted immediately if C_k and C_k+1 straddle a Target 40 Adversarial Critic blocker (40_critic_review.md), a Target 70 CI/Bot rejection, or a human Socratic nudge (N_nudge).
  • Stage 2 (Pair-Level Semantic Auditor): Stage 1 is strictly a high-recall candidate generator (eliminating ~90% of commit history in milliseconds). The surviving (C_k, C_k+1) pairs are handed to the Stage-2 Auditor, which asks: "Is C_k+1 a stylistic refactor of working code, or did C_k step on a hidden invariant that C_k+1 had to undo?"

2. Preventing "Lazy Retractions" (Reward-Hacking Vector 3's !Counter-Offer)

We did worry that giving a worker subagent an explicit constitutional right to revoke its !Promise (The Circuit Breaker) might create a lazy bail-out attractor whenever a ticket requires tedious multi-package refactoring.

We prevent reward-hacking on !Counter-Offer via a strict Constructive Witness Receipt (SCAR-PROC-150):

  • Tedium is Never a Valid Counter-Offer: A worker cannot issue a !Counter-Offer with prose claims like "This blast radius is too large" or "This touches 14 files across 3 packages." If the supervisor sees a prose-only retraction, it rejects the retraction immediately.
  • The Required Witness Receipt: To legally invoke Vector 3's !Counter-Offer and halt a target, the subagent must emit a machine-verifiable contradiction proof containing at least one of:
    1. A Mutually Exclusive Specification Proof: Quoting two active requirements (Req A at file:line vs. Req B in the ticket/test suite) whose post-conditions are logically unsatisfiable (P ∧ ¬P), accompanied by a minimal reproduction test demonstrating the collision.
    2. A Compiler / Public-API Lock: Exact dart analyze / compiler diagnostic output proving that satisfying the ticket inside the allowed package boundary violates an upstream sealed type or visibility boundary that the ticket forbids modifying.
    3. A 2-Option Architectural Fork (Option A vs. Option B Trade-Off Matrix): When a hidden precondition surfaces mid-flight, the subagent cannot simply stop—it must present the exact file:line blast radius of both architectural paths so the supervisor (or human at the pause gate) makes an N >= 2 decision.

Because constructing a valid falsification receipt actually requires more precision than doing normal multi-file work, lazy reward-hacking collapses to zero.


3. Expressing Async Teardown & Late-Arrival Races in SCAR-PROC-152 (strict digraph)

You put your finger on the exact boundary between structural graph topology and temporal interleaving!

In pure static Graphviz DOT (Target 15), a standard state node [Fetching] with outgoing edges to [Success], [Error], and [Disposed] satisfies out-degree >= 2, which catches forgotten teardown/cancel paths—but by itself, a plain DiGraph doesn't stop a zombie async callback (await repository.fetch()) from resuming after the state machine has already traversed the dispose edge into [Disposed].

Here is how we bridged that gap across Target 15 (strict digraph) and Target 40 (Critic):

  1. In the Target 15 strict digraph (SCAR-ASYNC-01 / Late-Arrival Edge Requirement): Any state node with an asynchronous in-flight operation (shape=parallelogram or async=true attribute in our DOT convention, such as [AwaitingNetwork] or [StreamSubscribed]) is forbidden from treating [Disposed] or [Cancelled] as a passive sink. The graph must explicitly model late-arrival resolution edges out of the cancelled/disposed transition (or annotate the edge with the physical cancellation token / mounted / isClosed guard predicate on the transition label, for example: AwaitingNetwork -> StateEmitted [label="onResolve [guard: !isClosed && reqId == activeReqId]"]).
  2. Forcing the Token Sequence Before Code Generation: Even without a full TLA+ model checker running on the DOT file, forcing the planner at Target 15 to write the explicit [guard: !isClosed && epoch == currentEpoch] edge attribute in the strict digraph primes the KV-cache before Gherkin generation (Target 16). Then Gherkin automatically unrolls a mandatory Late-Arrival Zombie Resolution Scenario (Given request A is in flight, When the controller is disposed/cancelled before A resolves, And request A subsequently completes, Then zero state emissions or listener notifications occur), which becomes a failing TDD test at Target 30 and is audited under Pillar 6 by the Target 40 Critic.

Thank you again for reading with this level of depth—your Scar Ossification / AST Context Signature formulation is going straight into the formal paper today!

Collapse
 
paullovelace profile image
Paul Lovelace •

The quota-vs-style-guide loop is the part I recognised instantly — burning a 7-day window in 3 days and watching the model ignore the same DON'T rules feel like two different frustrations until you put them side by side. The hot-stove line is doing a lot of work there. Have you actually seen a scar stop a repeat regression in your own log, or is the 238-ticket evidence still mostly the miner's retrospective view?

Collapse
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

Great question, Paul—and the answer is prospective, not retrospective: those 238 tickets (now up to 251 in FIELD_TEST_LOG.md as of today!) are forward-time production runs, and we actually watch the scars fire in real time inside the transcript.jsonl flight-recorder logs.

Because our independent auditor reads the raw .jsonl transcripts—including the model's internal thinking blocks—we don't just infer that a bug didn't happen; we literally see the moment a scar intercepts the model's train of thought.

Here are three concrete places where we watch scars stop repeat regressions in the live logs:

  1. Inside the Builder's thinking Trace (Turn-0 Deflection Before Tool Emission):
    In early tickets (FT-001 through FT-030), agents repeatedly stepped on async teardown rakes—like emitting state after an await gap when a controller or stream had already been disposed (SCAR-ASYNC-01). Once we rewrote that rule into the 4-Vector format (Wound -> Trap -> NOOO! Circuit Breaker -> Permanent Reflex) and bound it to Target 15/30, we started seeing thinking traces on subsequent tickets that literally read:

    "I was about to await repository.fetch() and emit the result directly, but that matches SCAR-ASYNC-01 (The Trap). Revoking that approach and guarding the post-await emission with an !isClosed check + adding a late-arrival disposal test."
    Across 200+ subsequent tickets, that failure class has a 0.0% repeat regression rate in committed code.

  2. Inside Target 40 (Pillar 6: Immune Defenses & Crash Antibodies):
    Every Target 40 Clean-Room Critic review has a mandatory 6th section (Pillar 6) where the un-primed Critic cross-examines branch.diff against the repository's active scar catalog. In dozens of field tests (FT-120 through FT-251), on the rare occasion where the Target 30 builder slips back into an old habit during a complex multi-file edit, the Target 40 Critic flags the exact SCAR-* ID as a BLOCKER inside the uncommitted working tree—killing the regression in 40 seconds before git commit (Target 50) can even unlock.

  3. Graduated 15ms Compiler Locks (SCAR-PROC-114 / package:synthetic_scars_lint):
    For scars that have undergone Scar Apoptosis (graduating from markdown into static AST lint rules or shell receipt checks), the interception is completely mechanical: when an agent emits a forbidden pattern, the analyzer trips in 15 milliseconds with the [SCAR-*] diagnostic and auto-fix recipe, and the agent repairs it on the very next turn with zero human intervention (N_nudge = 0).

(In fact, Part 5 of this series is dedicated entirely to those forward-time flight-test metrics and how Second-Order Scars get minted when the Target 40 Critic catches a builder shortcut during a zero-nudge run!)

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

The loop you describe is one I work against in Tokenade: each retry resends the whole history, so cost climbs with every turn of it. The cap belongs in the harness. A prompt rule is read by the same model that is stuck in the loop.

Collapse
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

"A prompt rule is read by the same model that is stuck in the loop."

That single sentence deserves to be framed on the wall of every AI engineering team!

You nailed the exact dual pathology of the retry loop:

  1. The Quadratic Cost Curve (what you tackle in Tokenade): Every failed turn resends the entire growing history (O(N^2) token billing), bloated with verbose tool payloads and duplicate file reads.
  2. The Self-Polluted Context Basin (Attention Dilution): Worse yet, resending that failed history fills the KV-cache with the model's own broken reasoning. By Retry 3, the tokens of the failed attempt outnumber the system prompt 20-to-1. Expecting a flat prompt rule ("Do not do X") read by that same hijacked model to break the loop is mathematically hopeless.

That is why we ended up pairing two orthogonal layers that complement what you are building with Tokenade:

  • In the Harness (The Make-Fork Cap): Our top-level supervisor (the Makefile DAG) never runs code edits or test loops in its own context window. It forks ephemeral subagents (exit 0) provisioned only with the phase-bound scars for that single target (|S_active| <= 5), bounded by physical shell/CI gates outside the model. When the worker exits, the retry history dies with it.
  • At Turn 0, Before the First Retry (4-Vector Scars): While the harness caps the loop after a collision, the 4-Vector Scar is designed to prevent Retry 1 from happening in the first place—by naming the exact high-probability sequence the model is about to emit (The Trap), handing it an explicit circuit breaker (The NOOO! / Epistemic Excuse) to revoke its own plan mid-reasoning, and supplying the concrete E-F-G-H replacement ladder (The Permanent Reflex).

Combining Turn-0 4-Vector Scars (preventing the trap before the first bad tool call) with Harness-level compaction and caps like Tokenade (stripping payload bloat and enforcing hard boundaries in the proxy/harness) is the exact defense-in-depth autonomous agents need. Going to check out Tokenade's Rust proxy architecture right now!

Collapse
 
rimonnassory profile image
Rimon Nassory •

Interesting approach! I particularly like the idea of turning recurring AI mistakes into reusable safeguards. We're building a context-aware AI architecture for Playwright test automation, and the combination of failure history, independent reviews, and deterministic checks is something worth exploring.

Collapse
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

Playwright test automation is an incredible domain for this architecture! End-to-end browser automation is notoriously packed with async flakiness, hydration timing mismatches, and brittle DOM selectors where an unscarred LLM will happily write tests that pass locally on a sunny day and flake at 3 AM in CI.

In test automation, three patterns translate directly from the framework:

  1. Flake Signatures as Negative Invariants (Wound -> Trap -> Reflex): LLMs love to write naive sleep timers (await page.waitForTimeout(3000)) or assert on transient UI states before microtasks settle. Capturing those flakiness post-mortems into Scars forces the generator to use deterministic locator predicates (await expect(locator).toBeVisible()) and network idle gates instead.

  2. The Clean-Room Critic on Test Assertions: A common failure mode when AI writes tests is the "Tautological Test" (asserting on mock data it constructed itself, or verifying an element exists without proving the user interaction actually triggered the backend state mutation). Having an independent Critic inspect the test diff before merge catches those hollow assertions cold.

  3. Mining Rapid-Regrets in Test Repos: If you look at Git commits in a Playwright repo where a developer touches a test file twice within 60 minutes, almost every single one is fixing an async race, an ambiguous CSS selector, or an unhandled modal overlay. That commit log is a goldmine for your initial scar registry.

Excited to see where your Playwright architecture goes—let me know if you run into interesting failure modes in browser automation!

Collapse
 
mp3sss31 profile image
mp3sss •

Linking quota burn and rule non-compliance to the same attention failure is a sharp observation — both are cases where an instruction exists but never materializes into an enforced consequence. The pain-feedback loop maps cleanly onto how desktop apps handle regressions: failing integration tests and crash-replay suites are the somatic memory of a codebase. Curious whether the scar-prompt injection is gated per-task or backed by a persistent lesson store that survives across sessions.

Collapse
 
randalschwartz profile image
Randal L. Schwartz Google Developer Experts •

Both — and the interplay between the two is the whole secret to why it scales to 600+ scars without blowing up the context window or suffering "lost-in-the-middle" attention dilution!

Specifically, it works just like Linux kernel modules (/lib/modules on disk vs. insmod into active RAM):

  1. The Persistent Backing Store Across Sessions (SCAR_REGISTRY.md + references/10_domain_vaccines.md):
    Every scar is permanently versioned in git inside the repository (scars/<category>/SCAR-XXX-NN.md compiled into SCAR_REGISTRY.md, currently 600 codified scars across 246 production tickets). When a run finishes at Target 90 (our post-mortem consolidation target), any new failure mode, Socratic nudge, or Critic blocker from that session is distilled into the 4-Vector structure (The Wound, The Trap, The Circuit Breaker, The Permanent Reflex) and committed to the repo so every future session and every agent (Gemini, Claude, subagents) inherits that somatic memory permanently.

  2. Phase-Bound & Domain-Gated Injection Per Task (insmod via Phi(s)):
    If you dump all 600 scars into Turn 1's system prompt, attention dilution (1/N softmax spread) kills compliance right back down to baseline. Instead, every scar in the persistent store carries an explicit Workflow Phase Binding Phi(s) (mapping it to specific Makefile targets Target 05 through Target 90) and a Domain/Package Tag (for example async, riverpod, jaspr, git, ci-triage).
    When the Depth-0 supervisor forks an ephemeral Level-1 worker subagent for a specific target — say Target 30 (Red-Green-Refactor TDD) on a jaspr web package — it queries the persistent store and injects only the subset { s in Scars | Target_30 in Phi(s) and domain(s) matches diff } (typically 5 to 12 surgical scars, ~1,500 tokens) directly into that worker's fresh prompt.

And I love your analogy: "failing integration tests and crash-replay suites are the somatic memory of a codebase." That is 100% true for code-level regressions. Where SCAR_REGISTRY.md completes the loop is capturing the regressions that cannot be expressed as a unit test in the app repo — like architectural patterns across brand-new classes (Segmently's context-recall study found 57 out of 70 engineering rules have no failing unit test!), git worktree contamination, or CI polling quota loops.

In fact, we just published Part 4.1 today, which dives straight into how we separate which scars belong in that local persistent repo store vs. which ones can be baked directly into model weights via DPO / LoRAs:
👉 dev.to/gde/all-that-glisters-is-so...