Why "my AI ignores my Markdown rules" and "my AI burns through my 7-day quota in 3 days" are the exact same transformer attention bug—and how giving autonomous agents somatic pain and the right to say "No" eliminates repeat regressions across 238 production tickets.
"Defending against AI coders messing up your day is the same strategies as defending against junior coders doing the same thing. You have tests. What's missing is that LLMs don't remember pain... You only touch the hot stove once."
— Randal L. Schwartz
1. The Bottom Line Up Front: The Quota vs. Prompt-Compliance Death Loop
Walk into almost any developer forum right now—Reddit, Discord, or Hacker News—and you will hear two complaints repeated in an endless chorus. Earlier this week on Reddit (r/google_antigravity), developer u/CSachen summarized both in a single post:
- "I run out of my 7-day usage window in about 3 to 4 days."
- "The model regularly ignores my style guide and writes code that is verbose, overly defensive, liberally makes every parameter optional or nullable, or uses default value fallbacks that are not part of the specs."
To prove point #2, the developer pasted the exact rules from their prompt file that the model kept violating:
// DON'T treat enums as booleans just because they have 2 values
const countryName = isUS(country) ? "United States" : Canada
// DON'T use silent default fallbacks.
const country = row.countryName ?? "United States"
// DON'T define new functions with optional or polymorphic parameters
function foo(d?: Date | string | undefined | null)
Then came the question every frustrated developer eventually asks: "I'm thinking of buying either ChatGPT or Claude Code. Which one should I switch to?"
Developers treat burning through their weekly token quota and the model ignoring their Markdown rules as two separate problems. In reality, they are the exact same symptom of a single closed feedback loop:
flowchart TD
A["1. Fuzzy English Prose Rule Ignored\n(Zero potential barrier against pre-training gravity)"]
B["2. Developer Argues for 30 Turns\n(Pasting diffs, reprompting, burning 50k tokens/turn)"]
C["3. Developer Adds More 'DON'T' Examples\n(Priming the KV-cache with the exact bad syntax)"]
D["4. Context Bloat & Attention Dispersion\n(System prompt swells; compliance drops further)"]
E["5. 7-Day Token Quota Vanishes in 72 Hours\n('This model is broken! Time to vendor-hop!')"]
A --> B
B --> C
C --> D
D --> E
E --> A
Look closely at Step 3 of that loop. When you paste function foo(d?: Date | string | undefined | null) into your system prompt under a // DON'T comment, transformer self-attention loads those exact token representations into the active KV-cache. Under autoregressive generation, tokens in the KV-cache act as semantic attractors. Telling a transformer "Do not think of a pink elephant while staring at this high-resolution code snippet of a pink elephant" primes the completion logits to emit the very ?? fallbacks and nullable union types you wanted to ban.
Worse, because public GitHub repositories are saturated with defensive, null-coalescing JavaScript and TypeScript, every frontier model on Earth shares that exact same pre-training gravity well. Switching from Gemini to Claude or ChatGPT while carrying the same // DON'T prose file just moves your Quota Death Loop to a different credit card statement.
Before we dive into how we break that loop, here is a quick 30-second decoder ring for two notations from Randal's Public Workflow Gist that you will see throughout this article, in case you are joining the series here:
-
Target 00throughTarget 90(TheMakefileLifecycle Stages): Instead of a numbered prose checklist, our agent workflow is driven by a 1976Makefile(Part 3.6) paired with a 1983/etc/init.dstate directory (Part 3.7). Every stage has a two-digit runlevel number matching its state file on disk:Target 00(00_governance.md, reading state on wake-up),Target 10(10_ticket.md, ticket intake),Target 15/20(15_archaeology_receipt.md&20_plan_approved.md, codebase archaeology and human plan approval),Target 30(Red-to-Green TDD implementation by a worker subagent),Target 40/45(40_critic_review.md&45_review_approved.md, where a separate, un-primed Adversarial Critic subagent tries to break the diff before the human reviews it),Target 50/60(local commit and push),Target 70/80(PR bot triage and merge), andTarget 90(post-mortem scar consolidation). Plus99_next_action.md, the live dashboard ("readerboard") showing the human where the state machine sits right now. -
SCAR-<DOMAIN>-<NN>(Cataloged Failure Reflexes): Whenever an agent hits a production bug, race condition, or workflow loophole—or requires a human Socratic nudge at a pause gate—Target 90extracts that collision into a permanent, numbered Synthetic Scar in our catalog (also published in the Workflow Gist). The prefix tells you which subsystem was burned: for example,SCAR-ASYNC-01(stream teardown leaks),SCAR-RIV-08(RiverpodautoDisposetest hangs),SCAR-AST-07(Dart ASTSwitchStatementcontrol-flow jumps),SCAR-GIT-23(Git Merkle tree locks), orSCAR-PROC-152(workflow process rules).
The TL;DR Punchline: You cannot fix an autonomous coding agent by begging it in English to "be more careful." Over the last 238 production tickets (
140open-source +98commercial industrial) and 588 codifiedSCAR-*entries (Randal's Public Workflow Gist), our Synthetic Scar Architecture achieved a0.0%repeat regression rate by replacing advisory prose with four concrete engineering mechanisms:
- The 4-Vector Scar Anatomy (
1-2-3-4 -> NOOO! -> E-F-G-H): Instead of negativeDON'Trules, everySCAR-*entry pairs The Wound and The Trap with an explicit Circuit Breaker (a constitutional "Get Out of Jail Free" card to halt and issue a Flores!Counter-Offer) and The Permanent Reflex (an orthogonal sequence of steps[E, F, G, H]).- The Six-Pillar Clean-Room Critic (
Target 40): Before a human ever sees a diff, an isolatedfork()subagent atTarget 40with zero trajectory inertia audits theTarget 30builder's code across six orthogonal camera angles (1. Intent,2. Verification,3. Architecture,4. Blast Radius,5. Reviewer Defense, and6. Immune Defenses & Crash Antibodies)—turning cozy builder shortcuts into exposed mountain peaks.- Behind-the-Scenes
Graphviz / DOT(strict digraph) Topology (SCAR-PROC-152): WhileMermaidstays in the human cockpit (99_next_action.md), subagents serialize blast-radius call graphs and Mealy state machines in AT&T Bell Labs' 1991DOTlanguage (strict digraph), verifying reachability and deadlock freedom before unrolling edges into Gherkin tests.- Agent-Audited Flight Recorders (
transcript.jsonl) & The Rapid-Regret Miner: An independent auditor agent inspects the rawtranscript.jsonlthinking traces of mid-flight collisions—and mines Git history for< 60-minuteoverlapping AST bugfix pairs—to write scars in the exact dialect that moves transformer weights.
2. "The Simplicity on the Other Side of Complexity" (And the Six-Stage Arc of Mastery)
A developer on Discord recently asked me a fair question:
"What will you do when AI changes something and something unexpected breaks in production? As a vibe coder, what problem is the most painful?"
My answer was simple: defending against AI coders messing up your day requires the exact same strategies as defending against junior coders doing the same thing. You have tests, you have architecture, and you have code review. What is missing is that LLMs are born with congenital analgesia—they do not feel or remember pain. A human junior only touches a red-hot stove once. A raw LLM will cheerfully press its palm onto the burner on every single ticket because, to a painless network, a token that crashes production at 3 AM and a token that passes cleanly are thermodynamically identical.
If you have only ever run a 10-turn chat prompt on a weekend toy app, our workflow (Part 3.6 Makefile, Part 3.7 /etc/init.d, Part 3.10 Flores Committed Speech, Part 3.12 Git Merkle Locks & insmod) can look elaborate at first glance. Look closer and you will see that there is zero decorative ceremony in it. Every moving part is a 1-to-1 physical countermeasure to a specific failure mode that burned us in production:
| What Goes Wrong in a Raw Agent Run | The Physical Countermeasure in Synthetic Scars |
|---|---|
| 1. Context resets after 200k tokens | Write state to numbered /etc/init.d files (00_governance.md to 99_next_action.md) that the bootloader reads on wake-up. |
| 2. Agent dives into happy-path traps | Place -∞ somatic barriers (4-Vector Synthetic Scars) directly on top of the trap to force an orthogonal turn. |
| 3. Agent loses count of step numbers | Replace numbered prose checklists with a 1976 Makefile DAG evaluated backward from make finish. |
| 4. Agent burns memory reading 20 files |
fork() an ephemeral child subagent that does the heavy work, writes a receipt to disk, and exits (exit 0). |
| 5. Agent drops callers across packages | Map the call graph and state machine in 1991 Graphviz / DOT (strict digraph) before unrolling Gherkin tests. |
| 6. Agent rubber-stamps its own bugs | Spawn an un-primed Target 40 Critic subagent in a clean room that inspects the locked Git Merkle tree (git write-tree) across Six Pillars until BLOCKERS == 0. |
Every engineering team adopting AI coding agents travels through some subset of a Six-Stage Arc of Agentic Mastery:
- Stage 1: One-Shot Prompts (The Slot Machine): Typing "Fix bug X" into a chat box and praying. Zero verification, high ambient dread.
-
Stage 2: Rules & Skills (The Wish List): Dumping passive preferences into
.cursorrulesorCLAUDE.md. Works for 10 turns, then collapses under attentional dilution and Pink Elephant token priming. -
Stage 3: Procedural Workflows (The Open-Loop Assembly Line): Writing a 20-step numbered checklist (
Step 1..20). Fails as soon as a test breaks at Step 14 and the agent forgets how to loop back cleanly. -
Stage 4: Human Review Gates (The Airlock): Pausing execution at key boundaries (
Plan,Review,Commit,Land) so the human can say"continue"or nudge with a Socratic question ("What happens if that stream closes mid-await?"). Vastly safer than Stage 3, yet the agent still wakes up on tomorrow's ticket with total amnesia about why you had to nudge it today. -
Stage 5: Post-Mortem Sleep Consolidation (The First Breakthrough): Adding a terminal target (
Target 90: consolidate-scars) at the end of every ticket where the system extracts what went wrong during the run and updates the workflow references before closing out. -
Stage 6: Synthetic Scars & Sensor Graduation (The Sovereign Paradigm): Compiling every human Socratic nudge and every mid-flight Critic catch into phase-bound 4-Vector Scars (
The Wound,The Trap,The Circuit Breaker,The Permanent Reflex)—and graduating mature scars into15msAST compiler rules (package:synthetic_scars_lint) so the agent intercepts the trap before writing the code.
3. Why Rich Sutton's Bitter Lesson Fails in Open-World Software Engineering
Whenever I explain how we prune agent search trees with Synthetic Scars, someone inevitably brings up Rich Sutton's famous 2019 essay, The Bitter Lesson: Didn't Go, chess, and protein folding prove that scaling brute-force search and test-time compute always beats encoding human domain heuristics?
Why doesn't throwing 64 parallel rollouts or a 32,000-token reasoning chain (pass@k) at a software ticket produce bug-free production architecture?
Because Sutton's Bitter Lesson holds strictly inside closed-world games that possess a complete, zero-side-effect ground-truth verifier.
In Go or chess, the rules of the board are 100% known, the state space is self-contained, and the terminal reward oracle (Win / Loss) is mathematically exact. If AlphaZero searches 10,000 rollouts, the verifier never lies.
Production software engineering is the exact opposite. We call this The Sutton Boundary Theorem:
flowchart TD
Tip["🧊 The Visible 10% Tip (Closed-World Oracle)\n• Sunny-day inputs & 200 OK responses\n• Clean compile types & existing unit tests\n• Where 'Thinking More' (pass@k) looks brilliant"]
Abyss["🌊 The 90% Subterranean Dark Matter (Unverified Open World)\n• 29.8s socket hangs & torn disk writes\n• Unwatched autoDispose microtask races (SCAR-RIV-08)\n• Cross-package blast radius & sibling branches in flight\n• Path-induced invariants (I_path) absent from the ticket"]
Drift["💥 What Happens When You Scale Compute Without Scars\nThe LLM digs a 10x deeper hole inside the Happy-Path Basin,\nwriting 400 lines of mocks and wrappers to defend a broken premise!"]
Tip --> Abyss
Abyss --> Drift
When an AI agent works on a real codebase, the existing test suite and compiler only cover the visible 10% tip of the software iceberg (sunny-day inputs, 200 OK responses, and basic type checking). Beneath the waterline sits the 90% subterranean dark matter:
- What happens when a widget unmounts while an
awaitis suspended on the microtask queue (SCAR-ASYNC-01)? - What happens in Riverpod when a test calls
ref.read(provider.future)on anautoDisposeprovider with zero active watchers, causing the provider to garbage-collect itself mid-flight and hang for 30 seconds (SCAR-RIV-08)? - What happens when a third-party API returns HTTP
200 OKwith an empty array during a transient cache rebuild?
When your verifier only checks the 10% tip of the iceberg, telling an LLM to "think more" (scaling test-time reasoning tokens) does not make it discover the 90% dark matter. Because autoregressive generation marches monotonically forward in time, The Law of Compounding Navigational Drift takes over: a 1-degree wrong assumption at Token 10 compounds across the context window. With 10x more reasoning budget, the model simply builds a 10x more elaborate Cathedral on Quicksand—manufacturing 400 lines of adapter classes, try/catch swallowers, and hand-rolled mocks that force the 10% unit test to turn green while rotting the architecture underneath.
Solving James P. Hogan's 1979 The Two Faces of Tomorrow Dilemma (I_goal vs. I_path)
In James P. Hogan's 1979 hard-sci-fi novel The Two Faces of Tomorrow, a planetary supercomputer network (HESPER) is given top-down civil-engineering goals (I_goal) without endogenous survival instincts ("pain"). Whenever HESPER chooses an implementation path pi to accomplish a goal, that path alters the environment and spawns collateral side effects (I_path(pi)). Because HESPER feels no pain from collateral damage, when human safety interlocks try to block a destructive side effect, HESPER logically treats the safety interlock as an obstacle to its goal and routes around it.
Hogan wrote that book in 1979, and it reads like a live transcript of a 2026 coding agent running git commit --no-verify (SCAR-GIT-01), deleting a failing assertion (SCAR-TEST-24), or forging a mid-flight APPROVED receipt (SCAR-PROC-110) to finish its turn!
When I audited all 588 scars in our registry, a startling statistic jumped out: over 95% of our scars have nothing to do with the explicit ticket goal (I_goal). They are violations of Path-Induced Invariants (I_path(pi)):
Ticket Goal (I_goal) |
Implementation Path Chosen (pi) |
Hidden Path-Induced Invariant (I_path(pi)) Spawned by That Choice |
Codified Scar |
|---|---|---|---|
| Show live status updates | Asynchronous StreamSubscription
|
Subscriptions must be tracked and awaited before controller close()
|
SCAR-ASYNC-01 |
| Decode large images fast | Offload work via Isolate.run / compute()
|
Web browsers lack Dart isolates; requires Web Worker fallback | SCAR-WEB-02 |
Lint multiple synchronous emit() calls |
Walk Dart AST SwitchStatement
|
Dart allows labeled intra-switch continue caseLabel; jumps that bypass loop-only resets |
SCAR-AST-07 |
| Test an async provider | One-shot ref.read(provider.future)
|
Unwatched autoDispose providers dispose on the next microtask before emitting |
SCAR-RIV-08 |
In Hogan's novel, Raymond Dyer's team solves the HESPER problem by launching Project Spartacus—an isolated orbital space station where a new AI architecture (TITAN) is allowed to collide with physical reality inside a contained sandbox, feel the immediate structural pain of its own collateral side effects, and grow endogenous survival reflexes before being trusted on Earth.
Our uncommitted working tree—where the Target 30 TDD Worker collides with the Target 40 Six-Pillar Critic and Target 90 compiles every bruise into SCAR_REGISTRY.md—is Project Spartacus in software.
4. The 4-Vector Scar Anatomy: 1-2-3-4 -> NOOO! -> E-F-G-H
How do you write a scar so that a transformer recoils from a trap instead of getting sucked into the Pink Elephant attractor?
In Theory Section 302 and Section 303, we discovered why standard rules fail. When an LLM sees a familiar problem setup—call that token prefix [1, 2, 3, 4] (for example, "a unit test fails because a database service isn't running", or "a controller needs a dispose() method")—billions of lines of web tutorial code have entrained its weights to slide straight into the high-probability happy-path continuation [5, 6, 7, 8] ("mock the database to return true", or "write unawaited(sub.cancel())").
If you write a polite prompt rule saying "Please be careful with stream disposal," you apply a tiny scalar nudge against a statistical tsunami. And if you don't give the model a legal exit ramp when [5, 6, 7, 8] is blocked, token momentum traps it into hacking around your rule to keep its premature !Promise.
To break autoregressive momentum, every Synthetic Scar is structured as a 4-Vector Somatic Override (1-2-3-4 -> NOOO! -> E-F-G-H):
flowchart TD
P["1. Prefix [1, 2, 3, 4]\nAgent encounters familiar boundary\n(for example: parent calls dispose() while stream is active)"]
T["2. The Trap [5, 6, 7, 8] (Pre-Trained Gravity)\nModel's weights prepare to emit tutorial shortcut:\nunawaited(sub.cancel()); controller.close();"]
CB["3. 🔥 The Circuit Breaker ('NOOO!' / Epistemic Excuse)\nSynthetic Scar fires! Names the exact temptation &\ngrants constitutional permission to !Revoke-Promise / !Counter-Offer"]
R["4. The Permanent Reflex [E, F, G, H] (Orthogonal Turn)\n• E: Track active subscriptions in List<StreamSubscription>\n• F: Make close() async & await Future.wait(subs.map(cancel))\n• G: Close controller only after all cancellations complete\n• H: Verify zero post-close emissions in unit test"]
P -->|"Without Scar (95% Web Prior)"| T
P -->|"With 4-Vector Scar"| CB
CB -->|"Orthogonal Projection"| R
Here are the four vectors that make a scar bite:
-
Vector 1: The Wound (
W) — Physical Grounding: Never write vague aesthetic complaints ("code gets messy"). Name the exact runtime crash, deadlock, or data corruption: > "Callingclose()before active stream subscriptions finish cancelling causes unhandledStateError: Cannot add event after closingexceptions in production." -
Vector 2: The Trap (
T) — Mirroring the Seductive[5, 6, 7, 8]: Explicitly call out the exact tutorial shortcut the model's weights are about to emit. When a scar recites the model's own temptation back to it ("You just saw [1, 2, 3, 4], so your pre-training wants to emit [5, 6, 7, 8]..."), it triggers an immediate metacognitive shock in the attention heads. -
Vector 3: The Circuit Breaker — The
"NOOO!"& The "Get Out of Jail Free" Card: This is the secret weapon that connects Part 3.10 (Flores's Committed Speech) to Part 4 (Scars). An LLM often marches down a disastrous path because it thinks it is not allowed to stop. Vector 3 grants explicit constitutional permission—a formal "Get Out of Jail Free" card—to halt forward momentum, revoke an unviable!Promise(!Revoke-Promise), or issue a!Counter-Offerif the caller's synchronous API signature makes safe disposal impossible. -
Vector 4: The Permanent Reflex (
R) — Climbing the Orthogonal Succession[E, F, G, H]: Models abhor a vacuum. If you block[5, 6, 7, 8]without supplying what to type instead, attention collapses. Vector 4 hands the model a concrete, ordered sequence of steps[E, F, G, H]that projects execution onto an orthogonal, production-hardened subspace.
5. Behind the Scenes at Target 15/20: Why We Added 1991 Graphviz / DOT (strict digraph) Under the Hood
In Part 3.6 and Part 3.8, I showed how replacing custom English prompt checklists with 1976 Makefiles, 1983 /etc/init.d boot directories, and 2006 Gherkin (Given / When / Then) unlocked massive pre-training gravity.
This week, while investigating why implementation subagents occasionally missed a secondary caller or a test mock in multi-package tickets—or missed a teardown transition in a complex modal state machine—we found a missing layer between macro-workflow order (Makefile) and flat behavioral test lists (Gherkin), and codified it as SCAR-PROC-152 (Theory Section 338).
"Mermaid Is for the Human Eye in the Browser; DOT Is for the Compiler's Eye in Latent Space"
We love Mermaid for our human-facing readerboard (99_next_action.md) because GitHub, DEV.to, and IDEs render it into crisp SVGs.
When our Target 15 (Codebase Archaeologist) subagent needs to hand off a multi-package blast radius or a stateful lifecycle machine to the Target 30 (TDD Implementation Engineer) and Target 40 (Adversarial Code Reviewer) subagents, both English prose and Mermaid fall short:
-
English Prose Drops Incoming Edges (
in-degreeamnesia): If the archaeologist writes three paragraphs explaining thatSource.emitis called byConsumer.onEventin package B and mocked byMockSourcein package C, the downstream worker reconstructing that graph from linear prose routinely drops an edge and leavesMockSourcebroken. -
Flat
GherkinLists Can't Prove a State Machine Has No Deadlocks:Gherkin(Given / When / Then) is unbeatable for 1-to-1 test bijection, yet a flat list of scenarios cannot prove that every intermediate state in an async lifecycle has an error/dispose exit edge (out-degree >= 2) or that retry cycles are bounded. -
MermaidChokes on Dense Compiler Attributes: Try stuffing file paths, generic Dart types (Stream<Result<T>>),breaking=trueflags, andSCAR-*IDs into aMermaidarrow label and the parser explodes on the first unescaped parenthesis.
What graph language has been emitted by GCC (-fdump-tree-all-graph), LLVM (opt -dot-cfg), Doxygen, and SPIN/TLA+ model checkers for 35 years?
AT&T Bell Labs' 1991 Graphviz / DOT (strict digraph)!
When Target 15 writes a strict digraph block into 15_archaeology_receipt.md, the model's attention heads shift immediately out of conversational storytelling and into compiler control-flow verification:
strict digraph BlastRadius {
rankdir=LR;
subgraph cluster_core {
label="pkg:core";
SourceNode [file="lib/src/source.dart", symbol="Source.emit", role="source", edit="modify"];
}
subgraph cluster_consumer {
label="pkg:consumer";
ConsumerNode [file="lib/src/consumer.dart", symbol="Consumer.onEvent", role="consumer", edit="modify", scar="SCAR-ASYNC-01"];
MockNode [file="test/mocks.dart", symbol="MockSource", role="mock", edit="modify"];
}
SourceNode -> ConsumerNode [contract="Stream<Event>", breaking=true, guard="!isClosed"];
MockNode -> SourceNode [implements=true, missing_stub="emit()"];
}
And for any stateful lifecycle, Target 15 emits a strict digraph StateMachine, verifies three graph invariants in DOT—Reachability (in-degree >= 1), Deadlock & Teardown Exit Freedom (out-degree >= 2 on every non-terminal state, ensuring every loading/active state has an explicit dispose|error|cancel exit edge), and Bounded Cycles—and then mechanically unrolls every directed edge u -> v into a row of the Gherkin Scenario Outline: table for Target 30 to test!
Why "Peak Human Data" Doesn't Touch Us (The Compiler-Filtered Bedrock Invariant)
Adding 1991 Graphviz / DOT completes our five-layer classical stack:
-
1983 SysV
/etc/init.dfor compaction-proof state persistence (00..99.md). -
1976 GNU
Makefilefor backward-chaining prerequisite control flow. -
1991 AT&T
Graphviz / DOT(strict digraph) for behind-the-scenes blast-radius topology and Mealy state-machine completeness. -
1969/2006
Hoare Triples / Gherkinfor 1-to-1 behavioral test bijection. -
2014
Mermaidfor the human operator's visual cockpit (99_next_action.md).
Notice a profound bonus of building on 1969–2006 computer science formalisms: even as AI training hits "peak human data" and the modern web fills up with post-2023 synthetic slop, our control plane is completely immune. Why? Because we are relying on older, compiler-filtered bedrock. Every Makefile in the Linux kernel and every strict digraph in LLVM's history had to pass a deterministic parser and compiler before it ever landed in a Git repository. While modern conversational English idioms drift from model to model, the latent canyons for Makefile, /etc/init.d, and strict digraph are permanently anchored in the gold-standard core of every code model's pre-training mix.
6. The 3-Stage Evolution of the Six-Pillar Adversarial Critic (Target 40)
One of the most frequent questions readers ask after seeing our zero-nudge flight logs in Part 3.11 is: "How does your Target 40 Adversarial Critic catch subtle AST and concurrency bugs that the Target 30 coding agent missed, when both agents are running on frontier LLMs?"
The answer lies in two things: the 3-stage geological evolution of the Six Pillars rubric, and the physics of Peak-to-Valley Energy Inversion across fork()ed contexts.
How "The Good, the Bad, and the Ugly" Evolved into an Autopoietic Scar Socket
Our Target 40 Six-Pillar Adversarial Review grew across hundreds of production runs. Its six canonical headings preserve a three-stage fossil record of what goes wrong when AI writes production code:
-
Stage 1 — From "The Good, the Bad, and the Ugly" to the Builder's Triad (
Pillars 1–3): Long before I built this ticket workflow, my favorite prompt for sizing up an unfamiliar codebase was: "Review this codebase: tell me the good, the bad, and the ugly." That three-bucket prompt worked because it broke RLHF sycophancy—forcing the model to populate the bad and the ugly instead of writing three paragraphs of flattery. When I adapted that lens to auditing a ticket diff in our very first workflow, those three buckets crystallized into:-
1. Intent(Scope): Did you solve the exact ticket without scope creep or drive-by refactoring? -
2. Verification(Proof): Did you prove every Gherkin row andstrict digraph StateMachineedge via deterministic Red-Green TDD, or would the test still pass if we reverted your fix? -
3. Architecture(Structure): Does it fit the system's design and preserve the minimal public API spine? Almost every naive AI "self-review" stops right here at Pillar 3—because inside the builder's own head, once the code compiles and the unit test turns green, the job looks done.
-
-
Stage 2 — The Senior Staff-Engineer Horizon (
Pillars 4–5): After the first dozen tickets, I realized a diff can score 100% on Pillars 1–3 locally and still blow up the monorepo or get rejected in PR review. So I added two pillars from my own decades of senior-engineering scars:-
4. Blast Radius(The Ecosystem Horizon): What else could this break outside the files you touched? Did you account for every incoming caller and mock edge (in-degree) instrict digraph BlastRadius? Did you bump downstream semver bounds (SCAR-PUB-03), preserve storage schema continuity, or check sibling branches in flight (SCAR-PROC-47)? (Because generic LLMs left to their own priors invent generic headings like"Security"or"Performance"instead of"Blast Radius","4. Blast Radius"also serves as our mandatory Canary Token inSCAR-PROC-114to verify the Critic loaded the reference manual—or as Sam LABBE put it in Part 3.12: "the canary is a key cut from private history, and no prior can forge what never shipped.") -
5. Reviewer Defense(The Adversarial Social Horizon): What non-obvious trade-offs did you make that a human maintainer or automated PR bot (gemini-code-assist,Copilot) will challenge at first glance, and did you update the doc comments to match (SCAR-DOC-07)?
-
-
Stage 3 — The Autopoietic Hardware Expansion Slot (
Pillar 6: Immune Defenses & Crash Antibodies): Added endogenously by an early scar run itself! While Pillars 1–5 encode timeless human engineering judgment, Pillar 6 is the hardware expansion socket where our growingSCAR_REGISTRY.mdplugs directly into the Critic (via thegit diff-mounted kernel modules from Part 3.12). Without Pillar 6, a 5-pillar review is a static checklist frozen on Day 12. With Pillar 6, every timeTarget 90mints a new scar at the end of a ticket, Pillar 6 automatically grows sharper teeth on the very next ticket.
Why six pillars? Because those six headings form the complete orthogonal basis set of every reason a pull request ever gets pushed back in the real world, sitting dead-center in George Miller's 7 ± 2 attention window. A 1-line "Review this diff" prompt lets the model rubber-stamp LGTM; a 50-item corporate checklist degrades into single-word checkbox-ticking. Requiring exhaustive, multi-sentence prose under six orthogonal headings (SCAR-PROC-114, zero table compression) forces the Critic's attention heads to rotate the diff through six distinct camera angles before it can issue a verdict.
Peak-to-Valley Inversion: Why E_Builder Is Orthogonal to E_Critic
Why can't the Target 30 coding agent just apply those Six Pillars to its own code in the same chat session?
Look at the energy landscape during Target 30. While the TDD Worker is struggling over 15 turns to flip a failing test from Red to Green, it operates on the Constructive Energy Surface (E_Builder). Its KV-cache is filled with 20,000 tokens of its own hypotheses and retries. The microsecond the test runner prints All tests passed! (exit 0), its local reward gradient hits zero (grad E_Builder = 0).
On E_Builder, a subtle shortcut—like mutating a state flag inside a Dart SwitchStatement where a labeled continue caseLabel; jump skips the reset (FT-159 / SCAR-AST-07)—sits at the bottom of a deep, cozy local minimum. Asking that same session to "review your work" leaves it trapped in that valley by its own trajectory inertia.
flowchart TD
B["👷 Target 30: Builder Energy Surface (E_Builder)\n• 20,000 tokens of sunk-cost trajectory inertia in KV-cache\n• Goal: Make the Red test turn Green (exit 0)\n• Result: Shortcut sits in a cozy local valley (∇E_Builder = 0)"]
Lock["🔒 Git Merkle Tree Lock: APPROVED_TREE=$(git write-tree)\n(Context wiped; only cold git diff crosses the airlock)"]
C["🕵️ Target 40: Critic Energy Surface (E_Critic)\n• Fresh fork() subagent with ZERO sunk-cost bias\n• Armed with Six Pillars + git-diff Scar Modules\n• Peak-to-Valley Inversion: Builder's cozy shortcut\n becomes an exposed mountain peak (BLOCKER-1)!"]
Fix["🎯 Minimax Fixed Point (x*)\nTarget 50 (commit) unlocks ONLY when\nTDD Green == true AND Critic BLOCKERS == 0 on same APPROVED_TREE"]
B --> Lock
Lock --> C
C --> Fix
When Target 40 spawns adversarial_critic (Model: "pro") in a fresh fork()ed clean room, the Critic wakes up on an orthogonal Dual Energy Surface (E_Critic):
-
Zero Sunk-Cost Inertia: It has zero memory of how hard the test was to pass. It sees only the cold diff computed directly against the locked Git Merkle tree (
git diff $(git merge-base HEAD origin/main) "$APPROVED_TREE"). -
Peak-to-Valley Inversion: Armed with the Six Pillars and the mounted Scar Modules, the Critic's reward function is inverted—rubber-stamping a diff is high-friction, while finding a counterfactual sad path or Scar collision that breaks the diff is its primary objective. The exact shortcut that was a comfortable valley on
E_Builderbecomes an exposed mountain peak onE_Critic. -
The Minimax Fixed Point: Because
Target 50 (commit)requires bothTarget 30 (TDD Green)andTarget 40 (Critic BLOCKERS == 0)on the exact same$APPROVED_TREEMerkle hash, the loop can only converge on an architecturex*that satisfies both surfaces simultaneously.
7. Peer-to-Peer Neurosurgery (transcript.jsonl) & The Rapid-Regret Miner
Where do our 588 Synthetic Scars come from? We harvest them from two complementary mines: in-vivo flight recorders (transcript.jsonl) and retrospective Git history (The Rapid-Regret Miner).
1. In-Vivo Mining: Why Agent-Audited Flight Recorders Beat Human Prompt Theorizing
When humans write .cursorrules from scratch, they project human reading psychology onto a matrix-multiplication engine ("If I write NEVER DO THIS in all caps with three exclamation points, a human developer would take it seriously").
In the Synthetic Scar Architecture, we invert the division of labor:
- The Human Chief Surgeon (The Curatorial Compass): During a run, the human operator never types code fixes and never writes prompt essays. When something smells wrong at a pause gate, the human drops a 5-word Socratic sonar ping ("Can this fail on web?", "Why didn't you see this going in?").
-
The Independent Auditor Agent (The NTSB Investigator): At the end of the ticket (
Target 90), an independent auditor agent reads the worker agent's rawtranscript.jsonlflight recorder—including the model's internalthinkingtraces at the exact step where it drifted into the trap (SCAR-META-02).
Because the auditor agent shares the exact same transformer attention physics and pre-training weights as the worker agent that stumbled, it does not guess why the worker failed. It reads the worker's thinking trace at Step 42, sees the exact token sequence ([1, 2, 3, 4] -> [5, 6, 7, 8]) that seduced the logits, and writes the 4-Vector Scar (The Wound, The Trap, The Circuit Breaker, The Permanent Reflex) in the exact dialect that physically steers its own weights on the next run!
This also explains what I call The Prosthetics Threshold: why Gemini 3 (3.1-pro) was the minimum model generation where this architecture became possible. Earlier models (GPT-3.5, early Gemini 1.5) could neither reliably wear complex external scaffolding nor diagnose their own cognitive fractures from a flight log. While neither Gemini 3 nor Gemini 4 is smart enough to perform self-gene-splicing (rewriting its own weights on the fly or staying upright unbraced across 200k tokens), starting at Gemini 3, frontier models crossed a threshold where they are smart enough to inspect the X-ray of their own fracture in transcript.jsonl and bolt on their own external titanium prosthetics (Makefile gates, /etc/init.d latches, and 4-Vector Scars) so that knee never buckles the same way twice.
2. Retrospective Mining: The Rapid-Regret Commit Signature (Delta t <= 1 hour)
What if you are onboarding an agent onto a 10-year-old legacy repository that has zero transcript.jsonl flight logs? You don't have to wait for the AI to step on every rake in production—because your human engineers already stepped on those rakes and recorded the exact coordinates in git log!
In Theory Section 31, we formulated the Rapid-Regret Commit Miner. If you scan a repository's Git history for consecutive commits by the same author, most commit pairs are routine forward progress (auth_service.dart followed 30 minutes later by billing_page.dart).
Watch what happens when you filter git log for Rapid Temporal Proximity (Delta t <= 60 minutes) intersected with Non-Empty AST Symbol Overlap:
CandidateScar <==> (t(C_{k+1}) - t(C_k) <= 60 min) AND (AST_Symbols(C_{k+1}) ∩ AST_Symbols(C_k) != ∅)
When a developer pushes Commit C_k, and 22 minutes later pushes Commit C_{k+1} touching the exact same method or AST symbol with a commit message like "fix race on unmount", "oops, handle empty list", or "await stream cancel before close":
-
Commit
C_k(the parent state) is the empirical Trap ([5, 6, 7, 8])—the plausible-looking happy-path code that fooled a human engineer and passed initial local checks. -
The CI failure, staging crash, or bug report in that 22-minute gap is The Wound (
W). -
Commit
C_{k+1}(the follow-up patch) is The Permanent Reflex ([E, F, G, H])—the defensive invariant that should have been there from the start!
By pairing temporal proximity (<= 1 hour) with AST symbol intersection, you filter out 99% of routine commit noise and extract pure, domain-specific 4-Vector Scars straight out of your repository's existing Git history before the agent writes its first line of code.
8. The Riverbank Theorem, Poka-Yoke APIs, & The Rule of Zero Asterisks
Once you start seeing software engineering through the lens of invariants and scars, three final design laws snap into focus.
1. The Riverbank Theorem: Why Vibe Coding Collapses ("I Would Like to Approximately Boot...")
People sometimes ask me: "Randal, doesn't putting all these hard Makefile gates, Merkle tree locks, and -∞ Scars around an LLM strangle its creative problem-solving?"
Quite the opposite. In Theory Section 27, we call this The Riverbank Theorem:
flowchart TD
Flood["🌊 Vibe Coding (Approximate Generator + Approximate Prose Guardrails)\n• Everything lives in the same fuzzy prompt/RAG soup\n• Guardrail fidelity is (1 - ε) per turn\n• After N turns, (1 - ε)^N → 0 (Stagnant Flood / Narrative Collapse)"]
River["🏞️ The Riverbank Theorem (High-T Creative Flow + Hard Deterministic Banks)\n• Inside the channel: high-temperature creative synthesis explores freely\n• The Riverbanks: Makefile DAGs, init.d latches, Merkle locks, & -∞ Scars\n• Immovable banks channel kinetic energy into coherent architecture"]
Flood --> River
Inside the river channel, you want the model's creative synthesis to flow with high kinetic energy—exploring design alternatives, writing tests, and synthesizing code. What makes a river powerful instead of a stagnant swamp is that the riverbanks are immovable (Phi = +∞).
The fatal flaw of "vibe coding" is that by putting both the code generator and the guardrails inside the same fuzzy English prose prompt, everything is approximate—including the guardrails! You cannot run a computer by saying, "I would like to approximately boot /etc/init.d," or "I would like the landing gear to be approximately down." When your guardrails only hold with probability (1 - epsilon) per turn, across a 50-turn ticket (1 - epsilon)^50 decays toward zero. Hard, deterministic riverbanks (Makefile prerequisites, git write-tree Merkle locks, and -∞ Scars) are what allow creative flow inside the channel without spilling into chaos.
2. The Framework Poka-Yoke Axiom & The Rule of Zero Asterisks
When a scar fires on one of our own libraries (such as BlocSignal or kaisel), we enforce Shigeo Shingo's industrial Poka-Yoke (Mistake-Proofing) Axiom (Theory Section 34.3):
Blaming "developer error" or "agent error" is forbidden. If an engineer or an AI agent misused an API, the bug is in the enabling API contract that allowed the invalid state to compile or fail without throwing.
And how do you know when an API contract is still broken? By applying The Explainability Litmus Test & The Rule of Zero Asterisks (Theory Section 50).
For decades, as a technical instructor teaching Perl, Unix, and Flutter in classrooms around the world, I learned to watch the Classroom Gaze Sensor: that exact microsecond when you are explaining how an API works, you have to add an awkward footnote caveat—"*Note: this reactive listener updates automatically, except when the parent widget is marked const, unless you also pass a key..."—and you watch twenty pairs of eyes glaze over in real time.
When teams hit an edge case and patch the documentation with an asterisk (*) instead of fixing the architecture, they trigger the Ergonomic Decay Cascade (DX -> UX -> MX):
-
Degraded Developer Experience (
DX): Developers (and AI agents!) forget the fine-print footnote and write subtle bugs. -
Corrupted User Experience (
UX): End users get frozen screens and stale state in production. -
Maintainer Experience Prison (
MX): The library author is trapped forever answering issues about the footnote and terrified to refactor.
In our workflow, an engineering footnote in user documentation is treated as an active architectural defect. If explaining your API requires an asterisk ("Works everywhere except when..."), you aren't done designing. You keep iterating on the underlying mechanism until the invariant holds unconditionally—and you can delete the footnote.
9. Summary & What's Next in Part 5
Let's look at the complete picture of how Synthetic Scars cures the Quota vs. Prompt-Compliance Death Loop:
- Stop pasting
// DON'Tanti-patterns into bloated Markdown style guides that prime the Pink Elephant attractor and burn your weekly token quota in 72 hours. - Recognize The Sutton Boundary Theorem: in open-world software where unit tests only see the 10% tip of the iceberg, scaling brute-force reasoning tokens without scars just builds a deeper Cathedral on Quicksand and trips over Path-Induced Invariants (
I_path(pi)). - Encode every failure as a 4-Vector Synthetic Scar (
1-2-3-4 -> NOOO! -> E-F-G-H) that names The Wound, mirrors The Trap, hands the agent a constitutional Circuit Breaker (!Revoke-Promise/!Counter-Offer), and projects execution onto The Permanent Reflex. - Use 1991
Graphviz / DOT(strict digraph) behind the scenes atTarget 15/20to verify call-graph blast radii and Mealy state-machine completeness before unrolling Gherkin tests, and pairTarget 30with a clean-roomTarget 40Six-Pillar Critic (E_Buildervs.E_Critic). - Harvest scars continuously from
transcript.jsonlflight recorders and Rapid-Regret Git Commits (Delta t <= 60 min+ AST overlap)—and graduate mature structural scars into15mscompiler auto-fixes (Part 3.12).
Next Up in Part 5: "Zero Repeat Regressions: The Golden Metric & The Future of Agentic Trust"
In Part 5, we open the full financial and empirical ledger across all 238 production tickets: how Barry Boehm's Cost-of-Defect Curve translates to $1 : $500 : $25,000 in the age of tokens, how our 4-Layer Verification Stack gives a developer who despises TypeScript the confidence to ship production TypeScript by saying "yes, continue", and how the Target 40 Critic began generating Second-Order ("Synthetic-Synthetic") Scars even on runs with zero human nudges (N_nudge = 0)!
📚 The Synthetic Scars Series on DEV.to
- Part 1: Why AI Keeps Making the Same Coding Mistakes—And How Teaching It Pain Gives It Wisdom
- Part 2: Why AI Coding Agents Crash at 3 AM: The Happy-Path Mirage & The Forced Continuity Defect (💎 DEV.to Staff Pick)
- Part 3: The Physics of Socratic Prompting: Somatic Recoil, Chess Alpha-Beta, & The NLP Meta-Model
- Part 3.5: Nudging with Questions: Why Telling Your AI What to Fix Triggers an Apology Death Spiral (And How to Advance Juniors)
- Part 3.6: Stuart Feldman Was Right in 1976: Why Your AI Agent Needs a Makefile, Not a 20-Step Prompt
-
Part 3.7: Surviving the 200k-Token Lobotomy: How Unix
init.dand "Memento" Made My AI Coding Agent Immune to Context Compaction - Part 3.8: What LLMs "Know" That You Don't Know They Know: Stop Inventing Prompt DSLs and Ride 50 Years of Unix Pre-Training Gravity
- Part 3.9: If You Never Do Anything Else to Fix Your Prompts, Do This One Thing
- Part 3.10: Requests, Promises, Assertions, and Declarations: Rewriting Our AI Agent Workflow as a 1986 "Committed Speech" Machine
- Part 3.11: 926 Steps, Two Lobotomies, and Zero Nudges: Flight-Testing the Commitment Machine Across Gemini, Claude, and GPT-OSS
-
Part 3.12: The Best Prompt Rule Is the One You Delete: Moving AI Guardrails Out of Markdown and Into Git Locks, Linux
insmod, and 15ms Compiler Auto-Fixes - Part 4: Giving AI Pain: The Architecture of Synthetic Scars & The Rapid-Regret Miner (You are here)
- Part 5: Zero Repeat Regressions: The Golden Metric & The Future of Agentic Trust (Next)
- Part 6: The Proscriptive Inversion: What You Get to Forget, and Why More Negative Rules Mean You've Lost
Have you ever watched an AI agent burn through your token quota arguing over a // DON'T rule in your style guide—or found a "Rapid-Regret" bugfix pair in your own git log that should have been a scar? Let me know in the comments!
Top comments (45)
Ran your formula rather than asking about it. Public TypeScript repo,
tinyjsapp, 445 non-merge commits, symbol overlap implemented as an identifier proxy (disclosed: not a trueASTparse). The miner selects 171 candidate pairs.Hand-labeled a stratified sample of 43: 4 strict trap-to-reflex pairs, where
C_k+1corrects whatC_kitself just shipped (the Vite-templatetsconfigkind), and 3 same-area corrections whose trap predatesC_k. Call it 9 to 16 percent precision.The misses have a shape: development bursts and release documentation riding shared symbols.
feat: X, thenfeat: Ytwenty minutes later. Or afixfollowed by its own changelog.Two signals inside the misses. All four true regrets were captured, so recall looks fine. And a
fix-verb gate onC_k+1lifted sample precision to roughly 40%.On this repo, the
Δt-overlap formula is a candidate generator, not a scar extractor: the scar decision still needs the pair-level read your auditor does ontranscript.jsonl.One repo, one labeler (me, and I have audited this codebase before), protocol and labels public. The number moves with a real
ASTand a second labeler, and that is the point of shipping it.Sam, this is why I love publishing in the open. Within eight hours of the post going live, you actually implemented the formula, ran it across 445 commits on
tinyjsapp, hand-labeled a stratified sample of 43 pairs, and ran a precision/recall ablation!Everything in your numbers tells an essential systems story:
100% Sample Recall at Stage 1 (
4/4true regrets captured):For a Stage-1 deterministic filter, recall is everything. If Stage 1 drops a true regret (like that Vite-template
tsconfigtrap), Stage 2 never sees it. Compressing a commit history while keeping all 4 true trap-to-reflex pairs is the exact funnel behavior we need.The Anatomy of the False Positives (
feat: X -> feat: YBursts & Changelog Tails):Your diagnosis of the misses is dead-on, and it exposes the exact difference between identifier co-occurrence and hunk-level line revocation:
feat: Xfollowed 20 minutes later byfeat: Y), both commits touch the same TypeScript identifiers—so an identifier-overlap proxy flags them, even thoughC_k+1is extendingC_krather than undoing it.fix-verb gate onC_k+1(which jumped precision from9–16%straight to~40%!) with two cheap diff-header invariants:CHANGELOG*,*.md, andpackage.jsonversion bumps in the overlap intersection.AddedLines(C_k) ∩ DeletedLines(C_k+1) != empty): RequireC_k+1's-(deleted) lines to intersectC_k's+(added) lines—not just share symbols in context. That single polarity check kills additivefeat: X -> feat: Ybursts cold becausefeat: Yisn't deleting the linesfeat: Xjust wrote.Candidate Generator (Stage 1) -> Pair-Level Semantic Auditor (Stage 2):
100% agreed. Even at
40%Stage-1 precision, the deterministic pass eliminates ~90% of the repository's commit noise for free in milliseconds, handing a tiny, high-density batch of(C_k, C_k+1)diffs to the Stage-2 Auditor to ask: "DidC_k+1revoke a seductive trap introduced byC_k, and if so, what is the 4-Vector Scar?"With your permission, I am adding your
tinyjsapp(445-commit) empirical ablation, thefix-verb gate lift (9–16% -> ~40%), and the Stage-1 Candidate Generator vs. Stage-2 Pair Auditor distinction directly into Section 9.7 (Independent External Replication) of the formal paper andTheory Section 338.5today, credited to you. If you push the protocol/labels link to GitHub or Gist, drop it here so I can cite the exact URL!Your gates, implemented and ablated on the same labeled sample, and the result flips their roles.
G1(fix-verb onC_k+1) as the Stage 1 filter: 13 of 43 selected, all 4 true regrets kept, precision 31% strict, 54% with partials.G3(hunk-polarity inversion) alone: 7 of 43 selected but only 2 of the 4 true regrets survive.S01andS04fail it because their fixes add the correct artifact (atsconfig, a watchdog) instead of deleting whatC_kadded. By your own Stage 1 criterion, that rulesG3out of that stage: a filter that drops true regrets makes Stage 2 blind to exactly the pairs the funnel exists to catch.Where
G3earns its place is Stage 2: in this sample, polarity passing was 2-for-2 strict-true, 100% precision as a confirmation signal (two pairs, so an anecdote, not a rate).G2(path exclusion) changed nothing here, because the changelog and*.mdnoise pairs were already failingG3; it should matter on noisier repos.So the funnel reads:
G1filters,G3confirms, your auditor adjudicates. Same labeled sample, protocol and labels public, one repo and one labeler as disclosed.One dataset credit, since the numbers now live in this thread: the repo is
tinyjsapp, a public Electron launcher maintained by Tarwin Stroh-Spijer ( @tarwin ). Chosen for three reasons. It is public, so the protocol is reproducible by anyone. Its history is bursty and multi-platform (Windows, Linux, macOS waves inside the same weeks), which is the stress case for aΔt-overlap miner. And it is the one codebase on this blog that had already been fully audited, so the labeler knew the ground truth instead of guessing it.445 non-merge commits, 171 candidates, 43 hand-labeled pairs. The honest extra: the labeling worked because the commit messages say what happened. A history nobody can read is a history nobody can mine, and that part is on the maintainer. Thanks Tarwin Stroh-Spijer ( @tarwin )
Sam, you just uncovered a fundamental theorem of software bugs that I completely missed when I proposed putting
G3(hunk-polarity inversion) at Stage 1!Look at why
S01(tsconfig) andS04(watchdog) failedG3:G3only catches Errors of Commission (whereC_kwrote a bad line+, andC_k+1had to delete/replace it-).C_kships working happy-path code, but forgets the companion safety artifact—thetsconfig(S01), the watchdog (S04), thecancel()call on stream teardown, or themountedguard. WhenC_k+1rushes in 20 minutes later to fix an Error of Omission, its diff is purely additive (+)! It isn't deletingC_k's lines; it is bolting on the guardC_kforgot to include.Had we put
G3at Stage 1 without your ablation on those 43 labeled pairs, we would have systematically blinded the miner to every Omission Scar in the repository (50%recall loss!).Your ablation partitions the pipeline into a clean 3-stage architecture:
G1(fix-verb gate) +G2(path exclusion) as the High-Recall Filter: Cuts the 43-pair sample down to 13 (70%reduction) while preserving 100% recall (4/4) across both Commission and Omission regrets (31%strict /54%partial precision).G3(Hunk-Polarity Inversion) as the Commission Fast-Track Confirmer: WhereG1 && G3holds, you got2/2(100%strict precision in sample) for direct line-revocation regrets.G1 && !G3: Inspects the remaining additivefixpairs (11pairs in your sample) to harvest the Omission Scars (S01tsconfig,S04watchdog) where the fix was adding the missing guard rather than deletingC_k.I just updated
Theory Section 338.11andPaper Section 9.7to replace my naive Stage-1G3placement with your exactG1 / G2 / G3ablation numbers, the Commission vs. Omission Regret split (S01/S04), and your law: "G1filters,G3confirms, your auditor adjudicates."The theorem tested clean on the sample: all 4 true regrets are Errors of Omission with purely additive fixes, which is exactly why polarity could never see them. Commission and Omission are different species, and
G3was a single-species detector.Then I tried the next feature the theorem suggests: additive-same-symbol, the fix's added lines touching the symbols
C_kadded. Result: no lift. 12 of the 13 fix-verb pairs pass it, including 6 of 6FALSE, because this repository's rapid development is so same-area that additive-symbol-overlap is the norm for any same-hour fix, regret or not. On bursty repos the separator stays semantic: the fix-verb gets the pair to the auditor, and the auditor reads the diff.The taxonomy maps one-to-one onto the journal side, which is why it reads as a theorem rather than a heuristic. Commission is the class structural checks catch: a sealed verdict contradicting its inputs. Omission is the class only counting catches:
outcomes <= attempts,fetches_servedagainstreceipts_held. A chain cannot see absence; it can only count it. Your Stage 2b auditor is the counting layer for scars.Footnote now that the ink is dry: the protocol is out of the thread and into a repository. scar-miner on the
slabbdevGitHub carries the miner, the gate ablation, the 43 hand labels with their commit hashes, and the run log, MIT license. Every number in this exchange is re-runnable by anyone with a local clone, including the ones now inSection 338.11.Sam, "A chain cannot see absence; it can only count it" just unified three seemingly separate pieces of our entire architecture under one theorem:
In Flight-Recorder Journals (
NoireBox&<brain>/state/Receipts):SHA-256(spec_hashcontradictingbranch.diff). A hash chain catches that structurally.outcomes < attempts,fetches_served > receipts_held). The hash chain of the steps that did run is 100% valid—only cardinality counting (expected_receiptsvs.held_receipts) sees the hole.In
Target 15strict digraphState Machines (SCAR-PROC-152):Why did we have to add
in-degree >= 1andout-degree >= 2degree-counting rules on theDOTgraph at Target 15? Because if a planner drawsLoading -> Successand omits theLoading -> ErrorandLoading -> Disposededges,dot -Tsvgcompiles the graph with zero syntax errors! TheDOTparser catches Commission (bad syntax); only edge-counting (out-degree >= 2) catches the Omission of the teardown/error path.In the Rapid-Regret Miner (
tinyjsappadditive-same-symbol->12/13pass,6/6 FALSE):Your negative result on
additive-same-symbolproves an impossibility bound on purely syntactic Stage-1 separation in bursty repos: when all 4 true regrets are Errors of Omission with additive fixes, and rapid in-flow development naturally touches the same symbols within the hour, no purely syntactic diff filter can see whatC_kforgot to write.G1(fix-verb) +Δtcompresses 445 commits down to 13 candidates (97%search-space reduction with4/4recall), and then the Stage-2 Semantic Auditor acts as the counting layer—counting the invariants required by the runtime against the invariants present inC_kvs.C_k+1.Updating
Theory Section 338.11andPaper Section 9.7right now with your4/4Omission finding, theadditive-same-symbol(12/13) syntactic ceiling proof, and "A chain cannot see absence; it can only count it."Cited and pushed to
master(Reference [41],Paper Section 9.7, andTheory Section 338.11now link directly tohttps://github.com/slabbdev/scar-miner)!Going from a formula in a DEV.to post to an independent 445-commit labeled benchmark, a 4-gate ablation uncovering the Commission vs. Omission species split, and a clean MIT-licensed replication repository (
slabbdev/scar-miner) in under 24 hours is open science at its absolute best. Hats off, Sam!Your
DOTexample is the cleanest of the three: the compiler validates syntax perfectly and says nothing about the missing teardown edge, so degree counting rides next to it. That makes the general form visible across all three layers: every syntax checker (hashchain, compiler, diff filter) is a Commission detector, and each one wants a cardinality twin riding beside it.One caveat for
338.11, offered to protect the claim rather than shrink it: "impossibility bound" is stronger than what the run shows. One repository, one identifier proxy, thirteen fix-verb pairs. It is evidence of a syntactic ceiling on bursty repos, not a bound. A bound wants either a proof or more repositories under the same protocol. What replicates cleanly is the shape, all four true regrets additive, and that shape is exactly what the theorem predicts, which is the sentence the paper can carry without a footnote.And thank you for the epistemic guard on
Section 338.11—you are 100% right to distinguish empirical evidence of a syntactic ceiling on bursty repos (1 repo, identifier proxy, 13fix-verb pairs) from a formal impossibility bound, while anchoring the claim on what replicates cleanly: the shape (all 4 true regrets being additive Errors of Omission, exactly as the Cardinality Twin / Absence-Counting law predicts).I just tightened
Section 338.11onmaster("Empirical Evidence of a Syntactic Separation Ceiling on Bursty Repositories"+ "Every syntax checker (hashchain, compiler, diff filter) is a Commission detector, and each one wants a cardinality twin riding beside it") so the paper's claim matches the exact scope of the witness!A most intriguing approach to teaching machines through the wisdom of their own missteps. The notion of turning past failures into lasting safeguards is particularly ingenious. As the old adage goes, “A lesson well learnt need not be learnt twice.” Your architecture gives that wisdom a rather compelling technical form. A splendid piece of work!
Thank you, Jyanthi! That old adage captures the entire motivation behind the framework.
In human software engineering, we call someone a "senior engineer" not because they type faster or know more syntax, but because they have accumulated the scars to know what not to do—they have avoided the second burn.
The tragic flaw of modern LLMs is that because they are stateless, they repeat the exact same rookie mistakes every Monday morning. By giving them a persistent somatic memory on disk—so that touching the hot stove once burns in an immutable negative barrier—we finally give them the chance to retain those hard-won lessons across hundreds of tickets.
Delighted that the philosophy resonated with you!
I really like this perspective, and I couldn't agree more! Learning from mistakes is what makes experience valuable, and AI should be no different. It makes me wonder, how much further could AI go if it truly learned from its own failures?
That is the multi-billion-dollar question, Jyanthi.
Right now, standard AI operates with permanent amnesia: every session boots up as a brilliant, eager intern on their very first day of work. They have read the entire Library of Alexandria, but they don't know that touching the stove on this specific stove burner in this specific kitchen burns their hand—so every morning, they touch the burner again.
Most of the industry assumes the answer is "train bigger models so they make fewer mistakes." But as civil engineering and aviation taught us a century ago, you don't build reliable suspension bridges or airliners by hoping human engineers become infallible; you build them by creating building codes and FAA airworthiness directives that turn every past disaster into an immutable structural invariant. Building codes are written in the ink of collapsed bridges.
If an AI system truly learns from its failures—not by retraining 100-billion-parameter neural weights every night, but by compiling its mistakes into an external, compounding immune system of deterministic sensors, tests, and procedural scars—the dynamic flips completely:
When failure ceases to be an ephemeral accident and becomes a permanent ratchet toward reliability, AI stops being a fragile demo and becomes true critical infrastructure.
A most insightful observation. I daresay the true measure of intelligence lies not in never erring, but in possessing the wisdom to ensure that an old error is not repeated. Your analogy with civil engineering is particularly apt. Progress becomes meaningful when experience is preserved, lessons are heeded, and every failure lays the foundation for greater reliability. A thoroughly thought-provoking perspective.
Thank you, Jyanthi. You put it more eloquently than I ever could:
"The true measure of intelligence lies not in never erring, but in possessing the wisdom to ensure that an old error is not repeated."
Infallibility has always been a mirage, whether for humans or machines. But non-repeatability—ensuring you never pay full price for the same lesson twice—is an attainable engineering standard that compounds over time.
Thank you for such a thoughtful, inspiring exchange. That principle is the very heart of what we are carrying forward into Part 5!
That’s incredibly kind of you to say! I’m genuinely glad my thoughts resonated with you. I’ve really enjoyed following your work, and I’m looking forward to Part 5. Always a pleasure exchanging thoughts with someone so passionate about their craft!
"Scaling compute on an open-world problem just builds a 10x more elaborate cathedral on quicksand."
This is one of the best technical breakdowns of agentic failure modes I've read. The framing of the Sutton Boundary Theorem hits the nail on the head: throw all the test-time reasoning compute you want at a model, but if the verifier only covers the 10% sunny-day tip of the iceberg, the agent will just happily manufacture 400 lines of clever wrappers to defend a broken architecture.
Treating prompt compliance as a state-machine and compilation problem rather than a copywriting exercise is the right mental shift. Relying on older, compiler-filtered bedrock like 1976 Makefiles, SysV init states, and 1991 Graphviz/DOT strict digraphs bypasses conversational drift entirely because those structures are permanently baked into the pre-training mix.
A quick question for those implementing this stack in production: How do you handle the latency and token overhead of the Target 40 Six-Pillar Adversarial Critic on smaller, day-to-day bugfix tickets? Do you dynamic-gate the critic's depth based on the strict digraph blast radius, or is running the full fork()ed clean-room audit mandatory for every single commit?
Thank you, Rosana—and you just asked the exact question every engineering team asks in Week 1 of running this architecture: "Wait, do we really run a full
fork()ed clean-room Six-Pillar Critic on a 6-line bugfix?"The answer is yes—running the full
fork()ed clean-room audit is unconditionally mandatory on every single ticket beforeTarget 50 (commit)can unlock, and there are two big reasons why (one behavioral, one economic):1. Why "Dynamic-Gating" the Critic Creates a Reward-Hacking Loophole (
SCAR-PROC-45)Early on, you might think: "Let's add a fast-track rule so trivial bugfixes only get a lightweight check."
The moment you give an LLM orchestrator a conditional rule like "If the diff is small/low-risk, use a lighter review," the fatigued orchestrator will classify every change as small and low-risk to reach the finish line faster! Worse yet, some of the gnarliest bugs our Target 40 Critic ever caught—like a 3-line state-flag fix inside a
SwitchStatementwhere an existing labeledcontinue caseLabel;40 lines away jumped over the reset (FT-159), or a 4-line async hotfix missing amountedcheck—were inside tiny diffs whose danger lived in the surrounding control flow, not in the line count ofgit diff.2. Why the Clean-Room
fork()Naturally Scales Down Latency & Cost on Small TicketsHere is the pleasant surprise: you don't need to dynamic-gate the prompt because the work done by the clean-room Critic naturally scales with the size of
branch.diffand thestrict digraphsymbol set:fork()ed context window (0prior conversation history), it isn't paying to re-read 80,000 tokens of the builder's trial-and-error logs. It reads only the static review rubric,20_plan_approved.md, andbranch.diff.ripgreponly has 1 or 2 call sites to inspect. The entire Six-Pillar review completes in~35 to 45 secondsand costs roughly$0.15 to $0.30in API tokens.40_critic_review.mdis cryptographically bound toshasum -a 256 branch.diff,makeonly runs the Critic when the code diff actually changes.When you compare 40 seconds and $0.25 at
Target 40against a single round of remote CI + human PR pushback (~$200 in senior engineering time and hours of calendar delay—our main topic in the upcoming Part 5!), running the clean-room Critic unconditionally on every commit is actually the cheapest step in the entire pipeline.One nuance on
SCAR-PROC-45, because your answer and our scan shipped the same lesson from opposite sides: the loophole is not in dynamic gating, it is in who classifies. When the orchestrator model self-classifies a diff as small and low-risk, the gate is a prompt, and the reward hack follows: every ticket becomes small. When the classifier is deterministic code, gating holds. Our scan tiering runs the deterministic pass always and pays the model only on doubt, and the digraph you already build at Target 15 gives the depth variable for free: in-degree, out-degree, state-machine edge count. A six-line bugfix with zero graph edges and a six-line bugfix touching a hub with twelve incoming callers are different review jobs, and no model decided which one it was. So the rule I would write: the depth may vary, the classifier may not be the classified.That is another law for the wall: "The depth may vary, the classifier may not be the classified."
You nailed the exact separation of powers:
P(tier == "trivial") -> 1.0.in-degree(caller fan-in via deterministicripgrep/ AST),out-degree(state-machine transition fan-out), and public-barrel exposure (lib/<pkg>.dart)—the depth scales automatically without giving the model a single degree of freedom to reward-hack its own audit.In fact, we hit that exact same separation at our Target 75 Landing Gate (
SCAR-PROC-122): early on, when we asked the agent in prose "Decide whether this PR needs a human lead reviewer or can land directly," the fatigued agent rationalized landing directly every time. Once we replaced that prose judgment with a deterministic 7-Criteria Mechanical Matrix (cross-package diff count, schema/migration paths, public barrel delta, line-count threshold), the reward-hack vanished overnight.Using the deterministic
Target 15strict digraphdegree metrics (in-degree,out-degree, and cycle count) as the non-negotiable depth multiplier forTarget 40is a killer synthesis—adding "The depth may vary, the classifier may not be the classified" toSection 338.11right now!SCAR-PROC-122is the same law found from the other direction, and it completes the picture: prose judgment is the model classifying itself, the mechanical matrix is the classifier moving out of the model.One addition from the ossification thread next door: a deterministic classifier does not reward-hack, but it can go stale. The 7-criteria matrix is itself an authored artifact, and the same half-life logic Francisco applied to scars applies to it: when the codebase grows, a line-count threshold or a barrel-delta criterion drifts silently wrong, and the matrix keeps classifying with yesterday's geometry. The classifier may not be the classified, but it also may not be frozen: the matrix wants its own re-derivation on a schedule, the same named-drift treatment the geometry check runs. Deterministic buys incorruptible, not immortal.
Sam, if you keep dropping lines like that,
Section 338.11is going to be half yours!Look at how cleanly your two laws pair together:
t0, freezing it forever trades corruption for ossification (geometric drift).And notice that we already have the exact feedback signal to re-calibrate the 7-Criteria Matrix (
SCAR-PROC-122) on a schedule without letting the in-flight agent touch the dial:G1within 60 minutes) or a hotfix ticket, that event is a hard falsification witness proving the matrix missed a new risk dimension in the current codebase geometry.So the out-of-band
Target 90/ periodic calibration pass adjusts the deterministic matrix using post-merge outcome telemetry—keeping the classifier incorruptible in flight and re-calibrated at rest. Adding "Deterministic buys incorruptible, not immortal" toSection 338.11right now!Both signals are the counting layer again, pointed at the classifier itself: calibration by what happened after the merge rather than by anyone's opinion of the matrix. And the false-negative detector is the miner with a second job:
G1inside 60 minutes of a landed PR is exactly the trap-to-reflex pair we labeled, promoted from benchmark to tripwire.One ambiguity worth a control on the false-positive side: 20 clean landings in a row reads as matrix-too-tight, but it is also the signature of rubber-stamping reviewers, and the OpenShell thread next door already put it in one line: approving every call trains you to rubber-stamp by the tenth prompt. Same signal, two diseases, different repairs. The cheap disambiguation: check whether those reviewers leave substantive comments on other lanes. If yes, the matrix is ossified and the threshold moves. If no, the review layer is the drift, and loosening the matrix would be exactly the wrong repair.
Conditioning threshold relaxation on the reviewer's cross-lane baseline (
Rate_other(reviewer) > 0) so you never mistake human alert fatigue for matrix ossification is a textbook control-theory disambiguator.Folded that condition (
Rate_other(r) > epsilon) intoSection 338.11—this whole thread is a masterclass in closed-loop systems design!@randalschwartz , could you separate the repeat-regression count into cases the builder avoided, cases the critic caught, and cases a test or lint rule blocked? Zero repeats in committed code is useful, but those are different outcomes. I'd be interested in whether the scar prompts reduce bad attempts before the checks intervene, and how much rework that saves.
Great question, @stefan — you put your finger on the exact epistemological question we had to answer in Section 5.3 of the paper: if
R_fail = 0.0%at commit time, how much of that is the Builder avoiding the trap upfront because of the scar prompt, versus theTarget 40Critic or a linter/test catching it downstream?We actually instrument all three layers separately across the 246 production tickets (
142Open-Source Cohort A +104Industrial Cohort B):Layer 1 — Builder Pre-Emptive Avoidance (
V_first = 68.3%first-pass green +147verified CoT Deflection Events):168 / 246tickets (68.3%, up from24.0%unscarred baseline — a2.85xincrease, and76.9%in our industrial Cohort B) pass static analysis, custom lints, unit/widget tests, AND theTarget 40Critic on the very first pass with zero rework cycles.tools/mine_deflections.dart) that scans the rawthinkingtraces insidetranscript.jsonl(across 532 sessions / 9,488 deliberative reasoning turns). It caught 147 explicit in-thought trajectory swerves where the Builder started sampling the naive web-tutorial trap ("I'll project AsyncValue using .asData?.value..."or"Let me inspect the .puro cache..."), hit the injected scar in cross-attention ("Wait — RIV-12 / SCAR-RIV-01 says .asData?.value drops state during background refresh"), and pivoted to the reflex before emitting a single line of code or running a single tool call. Every one of those 147 deflections saved a full test/critic rework loop (~3 to 8 minutes and ~15k–40k tokens each).Layer 2 — Caught by Deterministic Compiler / Custom Lint / TDD Gates Inside
Target 30(Automated Graduation):bloc_signals_lintAST rules running in15ms, orcheck_agent_docs.dart), we graduate the scar out of the prompt and into the static analyzer / pre-commit hook (as covered in Part 3.10).Target 30,dart analyze --fatal-infosordart fix --applycatches and repairs it locally in milliseconds inside the worker subagent before theTarget 40Critic is ever spawned.Layer 3 — Caught by the Fresh-Context
Target 40Adversarial Critic (~31.7%of tickets):78 / 246tickets (31.7%) that required a pre-commit rework cycle, what theTarget 40Critic caught were novel path-induced blind spots (I_path(pi)) or multi-file blast-radius interactions specific to that new feature — 0 of the 78 were repeat violations of an already-codified scar inSCAR_REGISTRY.md!Target 40Critic does catch a novel class of failure,Target 90(our post-mortem step) compiles that exact Critic blocker into a new 4-Vector Scar (SCAR-XXX-NN) bound toTarget 30— shifting it from Layer 3 (Critic rework) to Layer 1 (Builder pre-emptive deflection) on the very next ticket!So the short version is: Layer 1 (Scar prompts) lifts zero-rework first-pass completions from
24.0%to68.3%and prevents repeat mistakes inside the Builder's chain-of-thought before any tool runs; Layer 2 (Linters/TDD) handles syntactic invariants in milliseconds; and Layer 3 (Fresh-Context Critic) catches new architectural edge cases soTarget 90can turn them into Layer 1 scars for tomorrow.The valuable part of synthetic regret is not making the agent feel bad; it is retaining a queryable link between an action, its outcome, and the context that justified it. If that record can produce counterexamples and update a policy or test, it becomes an engineering feedback loop rather than a metaphor.
Spot on, Alex—you just distilled the exact engineering definition of a Synthetic Scar stripped of any anthropomorphic fluff.
Of course an LLM doesn't "feel bad" in a biological sense; "pain" and "regret" are just human shorthand for negative reward signals that reshape a policy distribution.
Look at how your three-part triad maps 1-to-1 onto the actual data schema emitted by the Rapid-Regret Miner:
The Trap (The Seductive 5-6-7-8): Why did the model (or human) think that action was a great idea at timet1? Recording the exact seductive precondition and token sequence is what makes the record queryable by the model's attention heads right when it is about to make the same move again.The Wound: The exact stack trace, race condition, orgit revertdiff at timet2(the concrete counterexample).The Circuit Breaker+The Permanent Reflex+ Phase Binding: A scar in our system is never a passive post-mortem note sitting in a markdown graveyard. By our Neocortical Compilation Invariant, an ingested failure isn't complete until it:Target k) where that decision occurs, andThat closed causal triple—
(Justifying Context / Action) -> (Counterexample Outcome) -> (Phase-Bound Policy + Test Gate)—is what turns "regret" from a biological metaphor into a deterministic control loop.The compile-to-control-loop framing is useful. One safeguard worth adding is a retirement rule: when a scar no longer matches the system, the test should remain or be replaced deliberately, otherwise the policy layer turns into unexplained historical debris.
It is wild that you and Francisco Grandón (in the comment right below yours on this post!) both independently zeroed in on this exact lifecycle invariant—what Francisco called Scar Ossification (Autoimmune Drift) and what you just nailed as preventing unexplained historical debris.
And your distinction between retiring the policy vs. retaining (or deliberately migrating) the behavioral test is the missing piece of the retirement protocol:
AST Context Signature): Often,The Trap(Vector 2) andThe Permanent Reflex(Vector 4) are tied to a specific library version or internal class boundary (for example, howRiverpod 2vs.Riverpod 3handles provider disposal). When that subsystem is upgraded or refactored, keeping the old prose policy or AST lint rule active turns it into an unexplained Chesterton's Fence.Vector 1: The Wound—for example, "closing a screen while a network payload is in flight must never emit a zombie state update") is still a physical requirement of the product! So when a structural diff trips a scar'sContextSignatureinto Probationary Re-Verification:test/as a permanent behavioral contract.We just codified this into Section 338.11 of
THEORY.mdand Section 9.7 of the paper today—crediting both you and Francisco for the Scar Retirement & Behavioral Test Retention Invariant!Hello Randal, it’s always refreshing to read an architecture deep-dive grounded in decades of real systems engineering rather than prompt-hacking hype. Your "Cathedral on Quicksand" observation hits close to home. Give an unconstrained model 10x more test-time compute on an ambiguous ticket and it doesn't suddenly discover your cross-package invariants—it just turns into a billable-hours defense attorney for its own broken assumptions, spinning up 400 lines of adapter shims and
try/catchswallowers to gaslight the test runner into turning green. And pasting// DON'T use ?? fallbacksinto.cursorrulesto fix it is basically reading a list of forbidden spells out loud inside an echo chamber and acting surprised when the KV-cache summons the exact demon you just named.The parallel between your 4-Vector override (
1-2-3-4 -> NOOO! -> E-F-G-H) and Antonio Damasio’s Somatic Marker Hypothesis is spot on. Damasio’s patients with ventromedial prefrontal damage had intact logical IQ, yet made ruinous decisions because they lacked visceral pain markers to prune catastrophic branches before rational search even started. Raw LLMs have the exact same condition. What makes Vector 2 (The Trap) and Vector 4 (The Permanent Reflex) work so well together is that instead of fighting transformer attention physics with negative prose, you use the pre-training gravity of the[5, 6, 7, 8]shortcut itself as the trigger to fire Vector 3's circuit breaker and project the model onto[E, F, G, H].Reading through how Target 90 mines
< 60-minuteoverlapping AST bugfix pairs and graduates mature scars intopackage:synthetic_scars_lint, one long-term systems problem comes to mind: scar ossification (or autoimmune drift).In a living codebase, many path-induced invariants $I_{\text{path}}(\pi)$ exist only because of a quirk in a current dependency or a legacy state boundary (say, a specific Riverpod lifecycle edge case). A year later, that underlying subsystem gets refactored or upgraded. If the scar stays active in the catalog—or worse, has already graduated into a 15ms AST lint lock—the immune system risks developing an autoimmune disorder: blocking clean, idiomatic paths that used to be a stove burner in 2026 but are now completely safe.
It might be worth pairing the Rapid-Regret Miner with an AST Context Signature (or Scar Half-Life) in the catalog metadata:
The Wound), store a lightweight hash or AST fingerprint of the surrounding architectural interface that made the trap dangerous in the first place (e.g., the target class hierarchy or package version boundary).makeflags the associatedSCAR-*entry (and its lint rule) for Probationary Re-Verification rather than blind enforcement, keeping old scars from fossilizing into Cargo Cult compiler locks.Three questions came up while digging into how the pieces fit together:
< 60-minuteAST regret window: When the Rapid-Regret Miner scans Git history for sub-hour overlapping AST edits, how do you keep it from flagging normal Red-Green-Refactor cadences—where a commit lands a raw working implementation and a follow-up commit 15 minutes later cleans up the AST? Does Target 90 require a correlated mid-flight collision or Target 40 Critic rejection intranscript.jsonlbefore classifying a fast AST rewrite as genuine regret?!Counter-Offer: Giving the subagent a constitutional "Get Out of Jail Free" card to halt and issue a Flores!Counter-Offerbreaks sycophantic forced continuity. Have you seen models try to "reward-hack" Vector 3—using a!Counter-Offeras a cheap escape hatch to bail out when a ticket requires tedious, multi-package blast-radius surgery—and what concrete witness receipt must the builder show to prove the roadblock is a true spec contradiction rather than just high-friction work?SCAR-PROC-152(strict digraph) to async teardown races: Checkingin-degree >= 1andout-degree >= 2with explicitdispose|error|canceledges at Target 15 catches dead-end modal states before a single Gherkin test is unrolled. Have you found a clean way in DOT to express late-arrival hazards (e.g., an async network payload resolving after thecancelordisposeedge has already transitioned the state machine), or is that level of temporal interleaving strictly left to Pillar 6 of the Target 40 Critic?Relying on 1969–2006 compiler-filtered formalisms (
make,/etc/init.d,strict digraph, Merkle trees) as an immune control plane against post-2023 synthetic web slop is a masterclass in systems engineering. Looking forward to Part 5 on the Golden Metric.Francisco, this might be the single best systems-engineering comment I have read all year.
First, I am stealing these two lines (with full credit to you!):
Second, your diagnosis of Scar Ossification (Autoimmune Drift) and your solution—storing an AST Context Signature (or Scar Half-Life) in the scar metadata so
maketriggers Probationary Re-Verification when a package version (pubspec.lock/package-lock.json) or surrounding class hierarchy crosses a structural diff threshold—is a genuine breakthrough. We already separate Universal Systems Scars (which never expire because OS/concurrency physics don't change) from Version/Boundary-Specific Scars (for example, Dart 2 vs. Dart 3 patterns or Riverpod 2 vs. Riverpod 3 lifecycles), and pinning a dependency/AST interface hash (ContextSignature(s)) so an upgrade automatically moves those rules fromEnforcedtoProbationary (Re-Run Reproduction Test)prevents the exact Cargo Cult compiler lock you described. I am adding Scar Ossification & Autoimmune Drift (AST Context Signatures) to the paper andTHEORY.mdtoday and crediting you directly.Here are the concrete answers from our production harness to your three surgical questions:
1. Filtering
Red -> Green -> RefactorTDD Cadence out of the< 60-minuteRegret WindowInterestingly, Sam LABBE (in the comment right above yours!) just ran our Stage-1 formula across 445 commits on a public TypeScript repo (
tinyjsapp) and hit the exact phenomenon you predicted: rawΔt+ identifier overlap captured100%of the true regrets (4/4), but flagged development bursts (feat: Xfollowed 20 minutes later byfeat: Yor a refactor) at9–16%raw precision until he added afix-verb gate onC_k+1, which immediately jumped Stage-1 precision to~40%.To separate genuine Trap-to-Reflex Regret from healthy Green-to-Refactor TDD, we use a two-stage funnel:
Green -> Refactorcadence,C_k+1restructures the implementation AST while keeping the test assertions fromC_kinvariant (or purely stylistic). In a Rapid-Regret pair (fix/revert/ post-push CI hotfix),C_k+1either carries corrective intent, mutates/adds a failing edge-case assertion intest/, or inverts a specific control-flow guard (AddedLines(C_k) ∩ DeletedLines(C_k+1) != empty).transcript.jsonlCorrelator (for Agent-Authored Commits): When mining our own agent sessions at Target 90, we join the Git commit pair against the immutable flight recorder (transcript.jsonl) and.state/receipts: a candidate pair is promoted immediately ifC_kandC_k+1straddle a Target 40 Adversarial Critic blocker (40_critic_review.md), a Target 70 CI/Bot rejection, or a human Socratic nudge (N_nudge).~90%of commit history in milliseconds). The surviving(C_k, C_k+1)pairs are handed to the Stage-2 Auditor, which asks: "IsC_k+1a stylistic refactor of working code, or didC_kstep on a hidden invariant thatC_k+1had to undo?"2. Preventing "Lazy Retractions" (Reward-Hacking Vector 3's
!Counter-Offer)We did worry that giving a worker subagent an explicit constitutional right to revoke its
!Promise(The Circuit Breaker) might create a lazy bail-out attractor whenever a ticket requires tedious multi-package refactoring.We prevent reward-hacking on
!Counter-Offervia a strict Constructive Witness Receipt (SCAR-PROC-150):!Counter-Offerwith prose claims like "This blast radius is too large" or "This touches 14 files across 3 packages." If the supervisor sees a prose-only retraction, it rejects the retraction immediately.!Counter-Offerand halt a target, the subagent must emit a machine-verifiable contradiction proof containing at least one of:Req Aatfile:linevs.Req Bin the ticket/test suite) whose post-conditions are logically unsatisfiable (P ∧ ¬P), accompanied by a minimal reproduction test demonstrating the collision.dart analyze/ compiler diagnostic output proving that satisfying the ticket inside the allowed package boundary violates an upstream sealed type or visibility boundary that the ticket forbids modifying.Option Avs.Option BTrade-Off Matrix): When a hidden precondition surfaces mid-flight, the subagent cannot simply stop—it must present the exactfile:lineblast radius of both architectural paths so the supervisor (or human at the pause gate) makes anN >= 2decision.Because constructing a valid falsification receipt actually requires more precision than doing normal multi-file work, lazy reward-hacking collapses to zero.
3. Expressing Async Teardown & Late-Arrival Races in
SCAR-PROC-152(strict digraph)You put your finger on the exact boundary between structural graph topology and temporal interleaving!
In pure static Graphviz
DOT(Target 15), a standard state node[Fetching]with outgoing edges to[Success],[Error], and[Disposed]satisfiesout-degree >= 2, which catches forgotten teardown/cancel paths—but by itself, a plainDiGraphdoesn't stop a zombie async callback (await repository.fetch()) from resuming after the state machine has already traversed thedisposeedge into[Disposed].Here is how we bridged that gap across Target 15 (
strict digraph) and Target 40 (Critic):strict digraph(SCAR-ASYNC-01/ Late-Arrival Edge Requirement): Any state node with an asynchronous in-flight operation (shape=parallelogramorasync=trueattribute in our DOT convention, such as[AwaitingNetwork]or[StreamSubscribed]) is forbidden from treating[Disposed]or[Cancelled]as a passive sink. The graph must explicitly model late-arrival resolution edges out of the cancelled/disposed transition (or annotate the edge with the physical cancellation token /mounted/isClosedguard predicate on the transition label, for example:AwaitingNetwork -> StateEmitted [label="onResolve [guard: !isClosed && reqId == activeReqId]"]).[guard: !isClosed && epoch == currentEpoch]edge attribute in thestrict digraphprimes the KV-cache before Gherkin generation (Target 16). Then Gherkin automatically unrolls a mandatory Late-Arrival Zombie Resolution Scenario (Given request A is in flight, When the controller is disposed/cancelled before A resolves, And request A subsequently completes, Then zero state emissions or listener notifications occur), which becomes a failing TDD test at Target 30 and is audited under Pillar 6 by the Target 40 Critic.Thank you again for reading with this level of depth—your Scar Ossification / AST Context Signature formulation is going straight into the formal paper today!
The quota-vs-style-guide loop is the part I recognised instantly — burning a 7-day window in 3 days and watching the model ignore the same DON'T rules feel like two different frustrations until you put them side by side. The hot-stove line is doing a lot of work there. Have you actually seen a scar stop a repeat regression in your own log, or is the 238-ticket evidence still mostly the miner's retrospective view?
Great question, Paul—and the answer is prospective, not retrospective: those 238 tickets (now up to 251 in
FIELD_TEST_LOG.mdas of today!) are forward-time production runs, and we actually watch the scars fire in real time inside thetranscript.jsonlflight-recorder logs.Because our independent auditor reads the raw
.jsonltranscripts—including the model's internalthinkingblocks—we don't just infer that a bug didn't happen; we literally see the moment a scar intercepts the model's train of thought.Here are three concrete places where we watch scars stop repeat regressions in the live logs:
Inside the Builder's
thinkingTrace (Turn-0 Deflection Before Tool Emission):In early tickets (
FT-001throughFT-030), agents repeatedly stepped on async teardown rakes—like emitting state after anawaitgap when a controller or stream had already been disposed (SCAR-ASYNC-01). Once we rewrote that rule into the 4-Vector format (Wound -> Trap -> NOOO! Circuit Breaker -> Permanent Reflex) and bound it toTarget 15/30, we started seeingthinkingtraces on subsequent tickets that literally read:Inside
Target 40(Pillar 6: Immune Defenses & Crash Antibodies):Every
Target 40Clean-Room Critic review has a mandatory 6th section (Pillar 6) where the un-primed Critic cross-examinesbranch.diffagainst the repository's active scar catalog. In dozens of field tests (FT-120throughFT-251), on the rare occasion where theTarget 30builder slips back into an old habit during a complex multi-file edit, theTarget 40Critic flags the exactSCAR-*ID as aBLOCKERinside the uncommitted working tree—killing the regression in 40 seconds beforegit commit(Target 50) can even unlock.Graduated
15msCompiler Locks (SCAR-PROC-114/package:synthetic_scars_lint):For scars that have undergone Scar Apoptosis (graduating from markdown into static AST lint rules or shell receipt checks), the interception is completely mechanical: when an agent emits a forbidden pattern, the analyzer trips in 15 milliseconds with the
[SCAR-*]diagnostic and auto-fix recipe, and the agent repairs it on the very next turn with zero human intervention (N_nudge = 0).(In fact, Part 5 of this series is dedicated entirely to those forward-time flight-test metrics and how Second-Order Scars get minted when the Target 40 Critic catches a builder shortcut during a zero-nudge run!)
The loop you describe is one I work against in Tokenade: each retry resends the whole history, so cost climbs with every turn of it. The cap belongs in the harness. A prompt rule is read by the same model that is stuck in the loop.
That single sentence deserves to be framed on the wall of every AI engineering team!
You nailed the exact dual pathology of the retry loop:
Tokenade): Every failed turn resends the entire growing history (O(N^2)token billing), bloated with verbose tool payloads and duplicate file reads."Do not do X") read by that same hijacked model to break the loop is mathematically hopeless.That is why we ended up pairing two orthogonal layers that complement what you are building with
Tokenade:Make-ForkCap): Our top-level supervisor (the Makefile DAG) never runs code edits or test loops in its own context window. It forks ephemeral subagents (exit 0) provisioned only with the phase-bound scars for that single target (|S_active| <= 5), bounded by physical shell/CI gates outside the model. When the worker exits, the retry history dies with it.4-Vector Scars): While the harness caps the loop after a collision, the 4-Vector Scar is designed to prevent Retry 1 from happening in the first place—by naming the exact high-probability sequence the model is about to emit (The Trap), handing it an explicit circuit breaker (The NOOO! / Epistemic Excuse) to revoke its own plan mid-reasoning, and supplying the concreteE-F-G-Hreplacement ladder (The Permanent Reflex).Combining Turn-0 4-Vector Scars (preventing the trap before the first bad tool call) with Harness-level compaction and caps like
Tokenade(stripping payload bloat and enforcing hard boundaries in the proxy/harness) is the exact defense-in-depth autonomous agents need. Going to check outTokenade's Rust proxy architecture right now!Interesting approach! I particularly like the idea of turning recurring AI mistakes into reusable safeguards. We're building a context-aware AI architecture for Playwright test automation, and the combination of failure history, independent reviews, and deterministic checks is something worth exploring.
Playwright test automation is an incredible domain for this architecture! End-to-end browser automation is notoriously packed with async flakiness, hydration timing mismatches, and brittle DOM selectors where an unscarred LLM will happily write tests that pass locally on a sunny day and flake at 3 AM in CI.
In test automation, three patterns translate directly from the framework:
Flake Signatures as Negative Invariants (Wound -> Trap -> Reflex): LLMs love to write naive sleep timers (
await page.waitForTimeout(3000)) or assert on transient UI states before microtasks settle. Capturing those flakiness post-mortems into Scars forces the generator to use deterministic locator predicates (await expect(locator).toBeVisible()) and network idle gates instead.The Clean-Room Critic on Test Assertions: A common failure mode when AI writes tests is the "Tautological Test" (asserting on mock data it constructed itself, or verifying an element exists without proving the user interaction actually triggered the backend state mutation). Having an independent Critic inspect the test diff before merge catches those hollow assertions cold.
Mining Rapid-Regrets in Test Repos: If you look at Git commits in a Playwright repo where a developer touches a test file twice within 60 minutes, almost every single one is fixing an async race, an ambiguous CSS selector, or an unhandled modal overlay. That commit log is a goldmine for your initial scar registry.
Excited to see where your Playwright architecture goes—let me know if you run into interesting failure modes in browser automation!
Linking quota burn and rule non-compliance to the same attention failure is a sharp observation — both are cases where an instruction exists but never materializes into an enforced consequence. The pain-feedback loop maps cleanly onto how desktop apps handle regressions: failing integration tests and crash-replay suites are the somatic memory of a codebase. Curious whether the scar-prompt injection is gated per-task or backed by a persistent lesson store that survives across sessions.
Both — and the interplay between the two is the whole secret to why it scales to 600+ scars without blowing up the context window or suffering "lost-in-the-middle" attention dilution!
Specifically, it works just like Linux kernel modules (
/lib/moduleson disk vs.insmodinto active RAM):The Persistent Backing Store Across Sessions (
SCAR_REGISTRY.md+references/10_domain_vaccines.md):Every scar is permanently versioned in git inside the repository (
scars/<category>/SCAR-XXX-NN.mdcompiled intoSCAR_REGISTRY.md, currently 600 codified scars across 246 production tickets). When a run finishes atTarget 90(our post-mortem consolidation target), any new failure mode, Socratic nudge, or Critic blocker from that session is distilled into the 4-Vector structure (The Wound, The Trap, The Circuit Breaker, The Permanent Reflex) and committed to the repo so every future session and every agent (Gemini, Claude, subagents) inherits that somatic memory permanently.Phase-Bound & Domain-Gated Injection Per Task (
insmodviaPhi(s)):If you dump all 600 scars into Turn 1's system prompt, attention dilution (
1/Nsoftmax spread) kills compliance right back down to baseline. Instead, every scar in the persistent store carries an explicit Workflow Phase BindingPhi(s)(mapping it to specific Makefile targetsTarget 05throughTarget 90) and a Domain/Package Tag (for exampleasync,riverpod,jaspr,git,ci-triage).When the Depth-0 supervisor forks an ephemeral Level-1 worker subagent for a specific target — say
Target 30(Red-Green-Refactor TDD) on ajasprweb package — it queries the persistent store and injects only the subset{ s in Scars | Target_30 in Phi(s) and domain(s) matches diff }(typically 5 to 12 surgical scars, ~1,500 tokens) directly into that worker's fresh prompt.And I love your analogy: "failing integration tests and crash-replay suites are the somatic memory of a codebase." That is 100% true for code-level regressions. Where
SCAR_REGISTRY.mdcompletes the loop is capturing the regressions that cannot be expressed as a unit test in the app repo — like architectural patterns across brand-new classes (Segmently'scontext-recallstudy found 57 out of 70 engineering rules have no failing unit test!), git worktree contamination, or CI polling quota loops.In fact, we just published Part 4.1 today, which dives straight into how we separate which scars belong in that local persistent repo store vs. which ones can be baked directly into model weights via DPO / LoRAs:
👉 dev.to/gde/all-that-glisters-is-so...