Lifecycle¶

This page is the precise order of events for each entry point. If you only read one diagram, read the Router one; it is the superset.

Agent.predict_batch
Agent.system_one / predict
Router.predict
Router.predict_batch
Model lifecycle
Caching with skip
Empty inputs
Ordering rules
Concurrency

Agent.predict_batch¶

predict_batch is the single implementation; system_one and predict call it with one state.

predict_batch(states, questions, batch_size=..., hooks=..., ...)
  │
  ├─ active  = default hooks + installed hooks + per-call hooks   (defaults first)
  ├─ ctx     = PredictContext(states, questions, model=self.model_id, agent=self)
  │
  ├─ try:
  │    │
  │    ├─ on_predict_start ─────────────────────────────┐
  │    │      a hook may:                                │
  │    │        • rewrite ctx.states / ctx.questions     │
  │    │        • set ctx.max_len / ctx.head_max_len     │
  │    │        • ctx.skip(results) ─────────────┐       │
  │    │        • raise (aborts; see errors)     │       │
  │    │                                         │      │
  │    ├─ if ctx.results is not None:  ◄─────────┘      │  cache hit
  │    │      skip tokenization and forward             │
  │    ├─ else:                                         │
  │    │      validate states is a list                 │
  │    │      for each batch chunk:                     │
  │    │        _encode_state ─► collate ─► _forward    │
  │    │      _decode_answers                           │
  │    │      ctx.results = [...]                       │
  │    │                                                │
  │    └─ (any failure here) ──► except BaseException:  │
  │              ctx.error = exc                        │
  │              on_error                               │
  │              re-raise                               │
  │                                                     │
  │   finally:                                          │
  │     ctx.elapsed_ms = now - ctx.started_at           │
  │     if ctx.results is not None:                     │
  │         ctx.usage = aggregate_usage(...)            │
  │         apply_confidence_gate(..., min_confidence)  │
  │     on_predict_end ─────────────────────────────────┘
  │
  └─ return ctx.results

The same ctx object flows through start, error and end, so run_id correlates them and an end hook can read ctx.error.

The finally guard is ctx.results is not None, not truthiness, and the empty batch is the only case where they differ: predict_batch([]) sets ctx.results = [] (laya/agent.py::Agent.predict_batch), so the guard passes, ctx.usage becomes {"input_tokens": 0, "output_tokens": 0}, and the confidence gate still runs over the (empty) result list. A call that raised before any result leaves ctx.results and ctx.usage both None. So for an end hook usage is None means "this call produced nothing", and it is not the same signal as "this call was empty" -- an audit hook that treats the second as the first reports every empty batch as a failure.

Agent.system_one / predict¶

system_one(state, questions, hooks=..., ...)
  └─ predict_batch([state], questions, hooks=..., ...)[0]

So system_one inherits every hook and the same lifecycle, with ctx.states == [state].

Router.predict¶

Router.predict(state, questions, model=..., hooks=..., on_predict_start=..., on_predict_end=...)
  │
  ├─ active = default hooks + installed hooks + per-call hooks
  │
  ├─ route(state, questions, ..., hooks=per-call, hooks_raise=...)
  │    │
  │    ├─ _route(...)                      detect script / language / workflow
  │    ├─ on_route  ──► ctx.decision       a hook may replace the decision
  │    └─ return ctx.decision
  │
  ├─ load(decision["model"])
  │    │
  │    ├─ already resident? ──► return it
  │    ├─ else build Agent(...) ──► on_load   (after the Router lock is released)
  │    └─ evict LRU checkpoints ──► on_evict  (after the Router lock is released)
  │
  ├─ ctx = PredictContext(states=[state], questions, decision, model=decision.model,
  │                       agent=agent, router=self)
  ├─ try:
  │    ├─ on_predict_start
  │    ├─ if ctx.results is None:
  │    │      result = agent.system_one(ctx.states[0], ctx.questions)
  │    │        └─ the Agent's own hooks run here (start / forward / end)
  │    │      result["routing"] = decision
  │    │      ctx.results = [result]
  │    └─ else:
  │           for each cached result: result.setdefault("routing", decision)
  │    └─ (any failure) ──► except: on_error, re-raise
  │    └─ finally: elapsed_ms, usage, on_predict_end
  │
  └─ return ctx.results[0]

Key points:

on_route runs before the model is loaded, so a hook can pin a checkpoint and avoid loading another one.
ctx.started_at is reset after route and load, immediately before on_predict_start, so the ctx.elapsed_ms an end hook reads is the prediction alone. Routing and a cold model load -- seconds, for a checkpoint that was not resident -- are outside it. Time the whole call from the caller, or from on_route, if that is what you need.
Router-level predict hooks wrap the whole call. They are not forwarded into the Agent; an attached Agent with its own hooks runs those too, which is expected.
A Router-level ctx.skip() still adds routing, so the return shape is stable.

Router.predict_batch¶

Each result is what predict returns for that request, so Router-level predict hooks run per request here too: every request gets its own PredictContext, run_id and elapsed_ms.

Router.predict_batch(requests, batch_size=..., hooks=...)
  │
  ├─ active = default hooks + installed hooks + per-call hooks   (defaults first; None and [] add nothing)
  ├─ route_batch(requests, hooks=per-call) ──► on_route, once per request   (no checkpoint loaded yet)
  │
  └─ for each checkpoint, in order of first appearance:
       │
       ├─ load(checkpoint) ──► on_load / on_evict
       ├─ for each request of this checkpoint, in input order:
       │      ctx = PredictContext(states=[state], questions, decision, model, agent, router,
       │                           max_len=request.get("max_len"),
       │                           head_max_len=request.get("head_max_len"))
       │      on_predict_start       a hook may redact, rewrite, set a token budget or skip
       ├─ group the requests left to infer by (questions, ctx.max_len, ctx.head_max_len)
       │      agent.predict_batch(states, questions, ...)  ──► one shared forward pass per group
       │      result["routing"] = decision;  ctx.results = [result]
       ├─ ctx.usage, once per request
       ├─ (any failure) ──► for every started request, in reverse input order:
       │                    ctx.error = exc, on_error, on_predict_end;  then re-raise
       └─ on_predict_end, once per request of this checkpoint, in reverse input order

Key points:

Every start hook of a checkpoint's requests runs before any of their end hooks, because they share forward passes. A cache that fills in on_predict_end therefore cannot serve a duplicate state within the same checkpoint group; it can across calls.
For the same reason the requests end in the reverse of the order they started, so a hook that sets something in start and resets it in end (a contextvars value, an OpenTelemetry context.attach / detach) unwinds to the value it found.
A start hook that replaces ctx.states, ctx.questions or the token budget changes its own request only: requests are grouped for the forward pass after their start hooks have run. Mutating a shared questions dict in place is different, and not what predict does: nothing of the group is inferred until all of its start hooks have run, so the change reaches every request that shares the dict, including those whose start hooks ran earlier, and the caller too. Assign a new dict to ctx.questions instead.
The checkpoint name is resolved before grouping, so an on_route hook that pins a request by an alias ("ml") shares that checkpoint's forward pass, and ctx.model is the resolved name.
A request may carry its own max_len / head_max_len, the per-request form of the token budget predict takes as call arguments. A start hook that sets ctx.max_len overrides it, because the hook runs after the context is built. Requests that ask for different budgets cannot share a forward pass, so a batch that mixes budgets makes one agent.predict_batch call per budget.
Every started request gets exactly one on_predict_end, even when another request's end hook raises; the first such error is raised after all of them have run.
If a checkpoint group fails, every one of its started requests fails with the exception: each gets on_error with ctx.error set to it, then on_predict_end. That includes a cache hit and a request whose question group had already run, because the caller gets the exception and no result for any of them, and it means ctx.error may be another request's failure (a start hook that raises for one request fails its group). Requests of checkpoint groups that already finished have ended with their results, as the earlier calls of [router.predict(...) for ...] would have.

Model lifecycle¶

on_load fires when a checkpoint is built; on_evict when one is freed. Both run after the Router's internal lock is released, so a hook may safely call back into the Router.

load("multilingual")
  │
  ├─ [lock]
  │    build Agent(...)          (seconds: download + weights)
  │    register in _agents / _order
  │    evict LRU if over max_loaded ──► evicted = ["english"]
  ├─ [unlock]
  ├─ on_evict("english")
  └─ on_load("multilingual")

unload("english")
  ├─ [lock] remove from _agents / _order
  ├─ [unlock]
  └─ on_evict("english")

attach(name, agent) registers an existing agent and does not fire on_load, because no checkpoint was built.

Caching with skip¶

on_predict_start
  ├─ cache hit?  ctx.skip([cached_result])
  │     └─ forward pass skipped
  │     └─ on_predict_end still runs
  │     └─ Router adds `routing` if missing
  └─ cache miss? nothing
        └─ forward pass runs
        └─ on_predict_end can store the result

See examples/hooks/cache.py for a working cache.

Empty inputs¶

Hooks still fire so audit sees every call:

input	`ctx.results` at end	`ctx.usage` at end
`predict_batch([])`	`[]`	`{"input_tokens": 0, "output_tokens": 0}`
`predict_batch(states, {})` (no questions)	one empty-answer payload per state	zeros, summed over those payloads
`system_one(state, {})`	a single empty-answer payload	`{"input_tokens": 0, "output_tokens": 0}`

No tokenization or forward pass happens in these cases, but on_predict_start and on_predict_end run. The third usage column is the point of the table: every one of these calls leaves an end hook with a real usage dict, because the guard that fills it is ctx.results is not None and [] passes it. Only a call that raised before producing results leaves ctx.usage as None.

Ordering rules¶

Process-wide default hooks run before installed hooks, and installed hooks run before per-call hooks, always.
Within a list, hooks run in list order.
For one event, every hook that implements it runs, in that order, before the next event.
on_error runs before on_predict_end on the failure path.
on_evict runs before on_load when a single load both evicts and builds.

defaults: [D]      installed: [A, B]      per-call: [C]
on_predict_start: D, A, B, C
on_predict_end:   D, A, B, C

Concurrency¶

Agent and Router are safe to call from many threads. Each call creates its own PredictContext, so contexts never leak across requests. The only shared state is the hook objects themselves, so a hook that is not thread-safe must either guard its own state or be installed with hooks_concurrent=False.

hooks_concurrent=True (default)      hooks_concurrent=False
  thread 1 ─┐                          thread 1 ─┐
  thread 2 ─┼─ hooks run in parallel   thread 2 ─┼─ one hook at a time
  thread 3 ─┘                          thread 3 ─┘   (RLock)

hooks_concurrent=False serialises each hook invocation, not whole calls: two calls can still interleave between events. It uses a re-entrant lock, so a hook may call back into the same Agent/Router without deadlocking.