DHP — Durable Handoff Protocol

Agent work that survives worker death. When a worker is kill -9'd, OOM-killed, or its spot instance is reclaimed mid-run, DHP hands the work to a standby that resumes from the last checkpoint — zero rework, zero lost results.

MCP standardized agent↔tool. A2A standardized agent↔agent. DHP is the missing layer: agent↔time — execution state that outlives any single worker.

pip install dhp-protocol

One-line install (pip + MCP registration + data dir):

curl -fsSL https://raw.githubusercontent.com/SIDDARTHAREDDY8/dhp/main/install.sh | bash

Direct into your agent:

Other ways to run: Docker (docker compose up -d for supervisor + transport server) · from source (git clone + pip install -e ".[dev]")

• The Problem
• What DHP Is
• Five Guarantees
• Quickstart
• How It Works
• The Kill Demo
• Architecture
• API Reference
• Measured Performance
• Comparison
• MCP & A2A Integration
• Roadmap
• Contributing
• License

You delegate a 20-minute task to an AI agent. At minute 14, the process is killed — OOM, spot reclaim, deploy, laptop lid. All 14 minutes of progress are gone. The agent restarts from zero, redoes everything, burns tokens twice.

Every agent framework today has the same hole: work is owned by the worker, not the handoff. When the worker dies, the work dies with it. Retries start from scratch. There is no primitive for "this unit of work exists independently of whoever is executing it right now."

DHP is that primitive.

DHP (Durable Handoff Protocol) separates work ownership from worker identity:

• Work is dispatched as a content-addressed handoff — a durable record of intent (task, inputs, output schema, budget).
• Workers claim handoffs, hold time-bounded leases, and checkpoint progress as they go.
• A supervisor watches leases. A silent worker is declared dead in ~6 seconds; its handoff is orphaned and a monotonic fence token is bumped — the dead worker's late writes are rejected, so two workers can never both believe they own the work.
• Any standby recovers the orphaned handoff from the last checkpoint. Zero rework.
• Checkpoints ship over TCP as they're made, so if the entire host dies, the peer already has everything.

It composes with the existing stack rather than replacing it:

1. No lost results — every completed handoff's result is durable and retrievable.
2. No crossed inputs — handoff IDs are content-addressed over intent; inputs cannot be swapped mid-flight.
3. No untyped results — completions are validated against a JSON Schema before acceptance.
4. No silent budget overrun — hard caps on steps, spend, wall-clock, and attempts. Exhausted work parks in a dead-letter queue: intact, visible, never silently dropped.
5. No runner lock-in — suspend → ship → resume on any host, any runner. The wire format is plain JSON-lines.

import dhp

store = dhp.Store("./demo-dhp")

# 1. Dispatch durable work
hid = dhp.dispatch(
    store,
    task_kind="demo.count",
    inputs={"target": 50},
    output_schema={"type": "object",
                   "properties": {"counted": {"type": "integer"}},
                   "required": ["counted"]},
)

# 2. Work it — the context manager heartbeats automatically in the background
with dhp.claim(store, hid, worker_id="w1") as ctx:
    state = dhp.last_checkpoint(store, hid) or {"n": 0}
    while state["n"] < 50:
        state["n"] += 1
        ctx.checkpoint(state)          # every step is a resume point
    ctx.complete({"counted": state["n"]})

Now kill the worker mid-run and watch recovery:

# terminal 1: supervisor watches leases, orphans the silent in ~7s
dhp-supervisor ./demo-dhp sup1 3600

# terminal 2: any standby picks up exactly where the dead worker stopped
with dhp.recover(store, hid, worker_id="w2") as ctx:
    state = dhp.last_checkpoint(store, hid)   # {"n": 20} — resumes here
    while state["n"] < 50:
        state["n"] += 1
        ctx.checkpoint(state)
    ctx.complete({"counted": state["n"]})

Steps already done are never recomputed. See QUICKSTART.md for the full 5-minute walkthrough.

A worker holds a handoff under a time-bounded lease (default: 2s heartbeat interval, 3 misses ≈ 6s TTL). Heartbeats run in a background thread — slow LLM reasoning between checkpoints never false-orphans a healthy worker.

When the supervisor sees an expired lease, it orphans the handoff: ownership moves to a tombstone holder and a monotonic fence token increments. Every subsequent write carries the worker's fence token; a stale worker's late checkpoint is rejected with LeaseLost. This is the same fencing pattern used by Chubby, ZooKeeper, and etcd — exactly-once ownership without distributed consensus.

Ten simultaneous recoverers → exactly one wins (atomic compare-and-set).

The handoff ID is a SHA-256 over the intent — task kind, inputs, output schema, budget — not the mutable state. A recovered handoff is verifiably the same work it was dispatched as. Tampered envelopes and checkpoints are rejected at ingest.

Checkpoints are appended to a JSON-lines log — one envelope header, then one line per checkpoint. Two transports:

• File (dhp.transport): shared directory or volume.
• TCP (dhp.net): TransportServer on the peer, NetShipper in the worker. Checkpoints arrive as they're made; the server verifies envelope identity and per-checkpoint hashes. The client also buffers locally, so a network partition loses nothing — it replays on reconnect.

Rework bound after any failure: ≤ 1 checkpoint.

All durability goes through the StoreBackend interface. The default is SQLite/WAL — zero dependencies, crash-safe, single-writer. Postgres (advisory locks + FOR UPDATE SKIP LOCKED give identical CAS semantics) is a clean extension, not a rewrite.

The demo that proves it — not a simulation:

python3 demo/run_mad_demo.py

What happens:

1. Host A: worker fetches 60 real Wikipedia pages, checkpointing + shipping each page over TCP to Host B as it's fetched.
2. At page 20: kill -9 on the worker. Real SIGKILL, mid-urlopen.
3. Supervisor orphans the handoff in 6.5s.
4. Host B ingests the 20 shipped checkpoints from its TCP log.
5. Standby on Host B resumes at page 20 — pages 1–20 are never refetched.
6. 60/60 complete. Real CSV dataset on disk.

=== MAD DEMO: 60 real pages, kill -9 at 20 ===
worker-A started (pid 4414)
*** kill -9 worker-A at 20 pages ***
supervisor orphaned after 6.5s
host B ingested: 20 checkpoints, 0 skipped
host B resumes from page 20 (worker-A died at 20)
worker-B started (pid 4444)
=== DONE: 60/60 pages, 60 fetched OK, zero refetched ===
MAD DEMO: GREEN

A network chaos harness (demo/net_chaos.py) SIGKILLs workers mid-ship across rounds: 24/24 checkpoints land on the peer, zero rework.

flowchart TB
    subgraph Interfaces
        MCP[dhp-mcp<br/>18 MCP tools]
        SDK[Python SDK<br/>dhp.dispatch/claim/recover]
        A2A[A2A bridge<br/>stable Task ID]
    end
    subgraph Core
        R[Runner<br/>lifecycle: dispatch → claim →<br/>checkpoint → complete]
        S[Supervisor<br/>lease watchdog → orphan<br/>leader election]
    end
    subgraph Durability
        SB[StoreBackend<br/>interface]
        SQ[(SQLite/WAL<br/>default)]
        PG[(Postgres<br/>extension)]
    end
    subgraph Portability
        NET[TCP transport<br/>live checkpoint shipping]
        LOG[JSON-lines log<br/>verified ingest]
    end
    MCP --> R
    SDK --> R
    A2A --> R
    R --> SB
    S --> SB
    SB --> SQ
    SB --> PG
    R --> NET
    NET --> LOG

Module map (dhp/):

import dhp

store = dhp.Store("./data")                        # or any StoreBackend

hid = dhp.dispatch(store, task_kind="...",          # create durable work
                   inputs={...},
                   output_schema={...},
                   budget={"max_attempts": 5})      # optional caps

with dhp.claim(store, hid, worker_id="w1") as ctx: # claim + auto-heartbeat
    state = dhp.last_checkpoint(store, hid) or {}   # resume point
    ctx.checkpoint(state)                           # per chunk
    ctx.heartbeat()                                 # manual (optional w/ ctx mgr)
    ctx.complete(result)                             # schema-validated
    ctx.release("reason")                            # cooperative handoff

with dhp.recover(store, hid, worker_id="w2") as ctx:# recover orphaned work
    ...

dhp.status(store, hid)                              # status/owner/checkpoints
dhp.approve(store, approval_id)                     # approve a gated action
dhp.deny(store, approval_id)                        # deny it (fail-closed)

Effect log — no duplicate side effects. Record each side effect by step ID; the first write wins, and the standby checks before re-firing:

with dhp.claim(store, hid, worker_id="w1") as ctx:
    if not ctx.effect_done("charge-card"):
        tx = charge_card(...)          # the side effect
        ctx.record_effect("charge-card", {"tx": tx})

If the worker dies between the charge and the next checkpoint, the standby sees effect_done("charge-card") and skips it. No double-charge.

Hard budget enforcement — no silent overrun. Declare caps at dispatch; the supervisor kills the handoff closed when they're hit:

hid = dhp.dispatch(store, ..., budget={"max_spend_usd": 5.0,
                                       "max_steps": 1000,
                                       "max_wallclock_s": 3600})
with dhp.claim(store, hid, worker_id="w1") as ctx:
    ctx.note_spend(0.42)   # real LLM API cost; supervisor fails closed on breach

Stall detection — catch livelocks, not just death. A worker can heartbeat forever while stuck. Report an activity digest; the supervisor orphans it as stalled when the digest repeats with no new checkpoint:

ctx.note_activity(sha256(tool_name + args))  # per tool call

Approval gates — mechanism, not prompt-as-policy. Destructive/financial actions need an external approval record. Fail-closed on deny or timeout:

try:
    ctx.request_approval("destructive", {"cmd": "drop table users"})
except ApprovalRequired as e:
    approval_id = str(e).split(": ")[-1]
# human/policy engine: dhp.approve(store, approval_id)
ctx.await_approval(approval_id)  # raises ApprovalDenied on deny/timeout

Not all failures deserve a retry. The supervisor classifies at orphan time:

• Lease expired (heartbeat timeout) → transient/infra → normal retry.
• Stalled (livelock) → bad plan → after max_stall_retries (default 2), DLQ instead of another blind retry. A transient failure resets the stall counter.
• Schema validation failure at complete() → DLQ with the typed error, not stuck "running".

hid = dhp.dispatch(store, ..., max_stall_retries=1)

Declare what the handoff may touch; everything else is denied fail-closed, and every check is audited:

hid = dhp.dispatch(store, ...,
                   capabilities={"network": ["api.stripe.com"],
                                 "filesystem": ["/tmp/work"]})

with dhp.claim(store, hid, worker_id="w1") as ctx:
    ctx.require_capability("network", "api.stripe.com")  # ok
    ctx.require_capability("network", "evil.example.com")  # CapabilityDenied

from dhp import TransportServer, NetShipper

# peer host:
TransportServer("./peer-logs", port=8471).serve_forever()

# worker host:
shipper = NetShipper("peer.example.com", 8471,
                     local_logpath="./ship-w1.log")  # partition buffer
shipper.ship_envelope(hid, envelope)
shipper.ship_checkpoint(hid, seq, state)             # verified server-side

dhp_dispatch · dhp_claim · dhp_checkpoint · dhp_heartbeat · dhp_complete · dhp_recover · dhp_status · dhp_last_checkpoint · dhp_health · dhp_record_effect · dhp_effect_done · dhp_note_spend · dhp_note_activity · dhp_request_approval · dhp_approval_status · dhp_approve · dhp_deny · dhp_require_capability

Not claims — chaos-tested numbers:

Verification: 59 pytest tests (lifecycle, concurrent CAS races, fuzzing, kill -9 integrity, network transport, v1.2/v1.3) · dhp-conform 10/10 · network chaos green.

DHP is not a workflow engine — it's the durability primitive workflow engines (and agent frameworks) can build on. If you run Temporal, keep it; DHP is for the agent layer Temporal doesn't reach.

MCP — run dhp-mcp --root <dir> and point any MCP client at it. The skill at skills/dhp-durable-work/SKILL.md teaches agents the dispatch → checkpoint → recover pattern.

A2A — dhp.a2a_bridge makes DHP the durability substrate under A2A tasks: the A2A Task ID stays stable while DHP swaps dead workers underneath (closing A2A's documented "no consensus or global transaction semantics" gap). See BRIDGE_DEMO.md: Agent B killed mid-task → Agent C recovered → client saw WORKING → COMPLETED, the swap invisible.

• Postgres StoreBackend implementation
• TLS for the TCP transport
• OpenTelemetry tracing across handoff attempts
• dhp dashboard — live handoff/lease/DLQ observability
• Multi-region supervisor quorum

PRs welcome. The bar: every behavior change needs a chaos test or conformance assertion proving it under kill -9, not just in the happy path.

git clone https://github.com/SIDDARTHAREDDY8/dhp
cd dhp
pip install -e ".[dev]"
python -m pytest tests/ -q   # 45 passed
dhp-conform                   # 10/10
python3 demo/run_mad_demo.py  # the kill demo

MIT — see LICENSE.