Router¶

laya.Router detects the language of each state and sends the request to the matching checkpoint, loading checkpoints on first use.

Router ¶

Router(
    models: Optional[Dict[str, str]] = None,
    device: Optional[str] = None,
    token: Optional[str] = None,
    revision: Optional[str] = None,
    revisions: Optional[Dict[str, Optional[str]]] = None,
    max_loaded: int = 2,
    default: str = "multilingual",
    auto_task_detection: bool = False,
    standalone_repos: bool = False,
    preload: bool = False,
    lang_guess: Optional[Any] = None,
    hooks=None,
    on_predict_start=None,
    on_predict_end=None,
    hooks_raise: bool = True,
    hooks_concurrent: bool = True,
    hooks_timeout: Optional[float] = None,
    agent_kwargs: Optional[Dict[str, Any]] = None,
    sha256_digests: Optional[
        Dict[str, Optional[Dict[str, str]]]
    ] = None,
)

Bases: HookRegistry

Lazily loads Laya checkpoints and sends each request to the right one.

from laya import Router

r = Router()
r.predict({"message": "Mein Konto wurde zweimal belastet"}, questions)   # -> multilingual
r.predict({"message": "I was charged twice"}, questions)                 # -> english
r.predict(state, questions, model="typed-decisions")                     # explicit

Models are downloaded and built on first use. max_loaded caps how many stay resident (least-recently-used is evicted), because all three together are ~1.16B parameters.

The default is 2, because automatic routing only ever chooses between english and multilingual: a cap of one rebuilds the checkpoint it just evicted on every script switch, which is seconds per request on exactly the traffic the Router exists for. Traffic that only ever sees one language never builds the second checkpoint, so the default costs it nothing. Lower it to 1 for a memory-constrained host, and raise it to 3 (or preload) when auto_task_detection, an explicit model= or an explicit task= can reach typed-decisions as well.

For a server or a demo, preload instead: a cold load costs seconds, while detection costs microseconds, so even the default still pays a load the first time a language appears.

r = Router(preload=True)                    # all three resident, routing is free
r = Router(preload=True, device="cuda")
r.preload(["english", "multilingual"])      # or just the two you serve

Hub revisions are opt-in. revision applies one commit to every model; revisions={"english": "...", "multilingual": "..."} overrides that per model, which is useful when standalone repositories were reviewed at different commits. Without either, huggingface_hub's normal default and existing offline cache are used.

A revisions entry that is None or blank is "no override for this model", so the model inherits revision -- the shape {"english": os.environ.get("EN_SHA")} writes when the variable is unset, which must not cost the caller the pin it did ask for. Nothing in revisions can unpin one model while revision pins the rest; leave revision unset and name the models you want pinned instead.

A blank revision is read the same way, which is a deliberate change in what the Router passes on: Router(revision=" ") used to reach resolve_revision, where a truthy but blank string suppressed the LAYA_REVISION fallback and left huggingface_hub's default, and is now dropped before it gets there, so a Router configured with whitespace behaves like one configured with nothing and $LAYA_REVISION applies. That is what "this configuration line was never filled in" has to mean if revision and a revisions entry are to be read the same way.

It is one of two places a weaker source ends up ahead of an explicit argument. The other is a per-checkpoint digest entry from LAYA_SHA256_DIGESTS, which wins over an expected_sha256 passed through agent_kwargs; see the class docstring.

Anything else laya.Agent accepts is reachable through agent_kwargs, which is merged into every checkpoint the Router builds:

Router(agent_kwargs={"lang_temperatures": {"de": {"temperature": [1.0, 1.4, 2.0]}}})
Router(agent_kwargs={"expected_sha256": {"model.safetensors": "a3f1..."}})
Router(agent_kwargs={"fast": True})

expected_sha256 there pins the same files on every checkpoint, which is what a single resident checkpoint or a shared tokenizer.json wants. It is never discarded wholesale by a checkpoint's own digests: where one also has an entry, from sha256_digests or from a per-checkpoint LAYA_SHA256_DIGESTS, the two maps are merged file by file, so a file only one of them names is still verified.

Two exceptions, both deliberate and both tested, because "merged file by file" is not the whole story and the difference is a supply-chain control:

A flat LAYA_SHA256_DIGESTS -- {artifact: digest} rather than {model: {...}} -- is not a layer here at all. verify_digests applies it itself, but only when nothing else pins (if expected is None), so ANY expected_sha256 reaching Agent, from here or from a checkpoint entry, means the flat variable is not consulted for that load. Verified on real files: a flat variable pinning model.safetensors plus an agent_kwargs map pinning tokenizer.json loads a tampered model.safetensors. Use the per-checkpoint shape, or name every file you care about in one map, if you need both. This is unchanged from main.
An explicit {} or None entry MASKS what would otherwise apply -- that is what "load this one unverified" has to mean, and test_an_explicit_none_entry_masks_a_flat_environment_map pins it.

And the precedence is per-checkpoint over shared regardless of where each came from, so a per-checkpoint entry synthesised from LAYA_SHA256_DIGESTS wins over an expected_sha256 passed here in code. An environment variable beating an explicit argument is worth stating plainly on a control like this; test_an_environment_pin_overrides_the_shared_one_per_checkpoint is where that is pinned.

For a file both name, the per-checkpoint entry wins. The two are not equally specific: the agent_kwargs map reaches every checkpoint the Router builds, and model.safetensors is the one name every checkpoint uses for a different file, so a shared entry for it cannot be a correct claim about all of them at once. Nothing raises over that overlap -- refusing it would reject a shared pin plus a per-checkpoint override, which is the ordinary shape and which loaded correctly before any of this existed. If you need to know which digest a checkpoint was verified against, read it back: the map handed to each Agent is the merge described above.

Both agent_kwargs and sha256_digests are public and mutable, and a checkpoint's entry is read on the load rather than at construction, so a pin assigned afterwards -- or added to an existing entry in place -- counts.

The names the Router sets for itself -- model_id_or_path, device, token, subfolder, revision and the hook arguments -- are refused here rather than silently shadowed, and the remaining names are checked against Agent.__init__ at construction, so a misspelled option fails on the Router(...) line instead of on the first request.

Artifact digests are opt-in and always per model: sha256_digests={"english": {...}} passes that {path relative to the checkpoint dir: hexdigest} map to the Agent that loads it, so a tampered or substituted weight file is refused before it is parsed. There is no Router-wide equivalent of revision because digests, unlike a commit SHA, are not shareable: the bundled repository ships a separate model.safetensors for each of english, multilingual and typed-decisions, so one flat map can only ever match one of them. A model listed with None or {} adds no files of its own, which loads it unverified unless agent_kwargs["expected_sha256"] pins it; it still masks a flat LAYA_SHA256_DIGESTS, which is what listing it that way is for.

The same split is available to a process configured only by environment: when LAYA_SHA256_DIGESTS holds a model-keyed map ({"english": {...}, "multilingual": {...}}) this seeds it per checkpoint, so a server that keeps several resident can pin each with its own digests instead of refusing to start on the second one. A flat LAYA_SHA256_DIGESTS keeps its existing meaning, applied by laya.revisions to every checkpoint the process loads, which is right for a single-checkpoint one. An argument entry wins over the environment for the model it names. A nested variable that names some checkpoints and not others says nothing about the others: they keep whatever agent_kwargs["expected_sha256"] pins them with, because pinning one checkpoint from the environment is not a request to stop verifying the rest.

Hooks are opt-in and run at the Router level: on_route sees the routing decision, on_load / on_evict see model lifecycle, and on_predict_start / on_predict_end wrap the whole route+infer call. See laya.hooks.

registered `property` ¶

registered: Dict[str, Dict[str, Any]]

The checkpoints registered beside the built-ins: name -> source and description.

Serialisable as it is.

loaded_revisions `property` ¶

loaded_revisions: Dict[str, Optional[str]]

Commit SHA each resident agent was loaded from (None for local paths).

resolve ¶

resolve(name: Any) -> str

Canonical key of a checkpoint this Router knows: built-in, aliased, or registered.

The instance counterpart of :func:normalise_name. Everything on the Router that takes a checkpoint name goes through here, so a checkpoint registered under papers is accepted wherever english is: model=, task=, load, unload, preload, revisions.

register ¶

register(
    name: str, source: Any, description: Optional[str] = None
) -> str

Add a checkpoint under name, to be served beside the built-in ones.

source is what laya.load accepts: a Hub repo id, a (repo, subfolder) pair, or a local directory holding model.safetensors and rl_agent_config.json -- a fine-tune exported by the training recipe, for instance. Nothing is downloaded or built here; the checkpoint loads on first use like the built-ins, is evicted and unloaded like them, and counts toward max_loaded like them.

description is free text about the checkpoint for whoever reads registered.

A name within one edit of a built-in is still registered, with a warning that names both and leaves the built-in unchanged.

A nested LAYA_SHA256_DIGESTS entry for this name is picked up here; an entry for a name the constructor did not know still fails at construction, as before.

Returns the canonical name. Registering an existing name replaces its source and unloads the resident Agent, so the next load builds from the new source.

unregister ¶

unregister(name: str) -> None

Remove a registered checkpoint: its source, description, revision and digest entries, and resident Agent.

A built-in name is refused (re-point it with register, it cannot be removed), and so is a name the router does not know. A resident Agent is unloaded first, waiting for an in-flight build of it, so a request that already holds it finishes; a request that names it afterwards gets the usual unknown-model error.

load ¶

load(name: str)

Return the Agent for name, downloading and building it on first use.

Concurrent callers share a single Agent instead of building duplicates.

attach ¶

attach(name: str, agent: Any)

Register an already-built Agent under name instead of loading a second copy.

Useful when the process has a checkpoint loaded for other reasons: a demo that already built convaiinnovations/laya can hand it to the router rather than pay for -- and hold in memory -- a duplicate 421M parameters.

A name the Router does not know yet is registered on the spot, with no source: the agent serves under that name while resident, and loading it again after an unload needs register(name, source) first.

preload ¶

preload(names: Optional[List[str]] = None)

Download and build checkpoints up front so no request ever pays a model load.

A cold load costs seconds; language detection costs microseconds. With every checkpoint resident, routing is effectively free -- which is what you want in a server or a demo. max_loaded is raised to fit both the requested checkpoints and all already-resident agents, so incremental preloading does not evict either.

unload ¶

unload(name: Optional[str] = None)

Free one model, or all of them.

route ¶

route(
    state: Union[str, dict, list, None],
    questions: Optional[Dict[str, Any]] = None,
    model: Optional[str] = None,
    task: Optional[str] = None,
    lang: Optional[str] = None,
    lang_guess: Optional[Any] = None,
    hooks=None,
    hooks_raise: Optional[bool] = None,
    hooks_timeout: Optional[float] = None,
) -> RouteDecision

Decide which checkpoint to use, then let on_route hooks observe or replace it.

ctx.decision is the RouteDecision; a hook may replace it (for example to pin a checkpoint) and the replacement is what gets returned and used. hooks are per-call hooks, appended after any installed on the Router.

predict ¶

predict(
    state: Union[str, dict, list],
    questions: Dict[str, Any],
    model: Optional[str] = None,
    task: Optional[str] = None,
    lang: Optional[str] = None,
    lang_guess: Optional[Any] = None,
    hooks=None,
    on_predict_start=None,
    on_predict_end=None,
    hooks_raise: Optional[bool] = None,
    hooks_timeout: Optional[float] = None,
    max_len: Optional[int] = None,
    head_max_len: Optional[int] = None,
    min_confidence: Optional[float] = None,
) -> Dict[str, Any]

Route, then answer every question in one forward pass on the chosen checkpoint.

The result is the usual system_one payload plus a routing key recording the decision. Router-level on_predict_start / on_predict_end hooks wrap the whole route+infer call and see ctx.decision; see laya.hooks. max_len / head_max_len override the agent token budget for this call (a start hook may set ctx.max_len / ctx.head_max_len).

predict_long ¶

predict_long(
    state: Union[str, dict, list],
    questions: Dict[str, Any],
    model: Optional[str] = None,
    task: Optional[str] = None,
    lang: Optional[str] = None,
    lang_guess: Optional[Any] = None,
    window: Optional[int] = None,
    stride: Optional[int] = None,
    aggregate: str = "auto",
    batch_size: Optional[int] = None,
    hooks=None,
    on_predict_start=None,
    on_predict_end=None,
    hooks_raise: Optional[bool] = None,
    hooks_timeout: Optional[float] = None,
) -> Dict[str, Any]

Route, then scan every window of the state instead of only its first one.

predict scores a state from a single window: anything past max_len is cut off (the first window, or for a conversation list the last) and never reaches the model. This routes exactly as predict does -- the same model/task/lang hints, the same router-level hooks, the same routing key and usage -- and scores the routed state with that agent's predict_long, which splits it into overlapping windows and aggregates per question. The aggregation rules are laya.agent.Agent.predict_long's: noul takes the strongest window, choice/score the most confident one.

Per-call hooks (hooks, on_predict_start, on_predict_end, hooks_raise, hooks_timeout) wrap the whole route+scan exactly as they wrap predict: the scan runs last, so a start hook that answers (ctx.skip(...)) or rewrites the state wins. max_len / head_max_len are not accepted here -- a window is sized by window or the checkpoint budget, and overriding the single-window truncation is what predict_long is for.

Parameters:

window (Optional[int], default: None ) –

state tokens per window. Defaults to the routed checkpoint's budget (max(64, max_len - head_max_len - 8); the 64 is a floor), capped at the room the questions leave for the state so no window is truncated again on the way in; a smaller window isolates a localized span.
stride (Optional[int], default: None ) –

token step between windows; defaults to half the effective window (50% overlap). A stride past that window is a ValueError, since the tokens between windows would reach no model.
aggregate (str, default: 'auto' ) –

"auto" (the per-type rules above) is the only mode.
batch_size (Optional[int], default: None ) –

cap on windows per forward pass, to bound memory on very long states.

Returns:

Dict[str, Any] –

The usual predict payload, with usage["windows"] counting the windows scored.

Raises:

TypeError –

the routed agent has no predict_long -- one attached by hand, since both Agent and ONNXAgent implement it -- so there is nothing to scan with. Always raised, whatever hooks_raise is, and so is every other error the scan itself raises -- none are caught: the scan is this method's own work, not a caller's hook, so the hook error policy does not decide whether it may be skipped. It previously ran as a start hook, where hooks_raise=False swallowed this and returned one window scored by system_one -- a different question than the one asked. An agent whose predict_long has no lang parameter is scanned without one and warned about, not failed; that is a signature check, not a swallowed error.

decide ¶

decide(
    state: Union[str, dict, list],
    schema: Any = None,
    *,
    questions: Optional[Dict[str, Any]] = None,
    return_details: bool = False,
    min_confidence: Optional[float] = None,
    **predict_kwargs,
) -> Any

Answer state against a schema (JSON schema or pydantic model) and return typed values.

See laya.structured. Pass exactly one of schema or questions; extra keyword arguments (for example model=, task=, hooks=) are forwarded to predict.

decide_batch ¶

decide_batch(
    states: Sequence[Any],
    schema: Any = None,
    *,
    questions: Optional[Dict[str, Any]] = None,
    return_details: bool = False,
    min_confidence: Optional[float] = None,
    **predict_kwargs,
) -> List[Any]

Answer many states against one schema (JSON schema or pydantic model) in one batched call.

The throughput form of :meth:decide: the schema is planned once and its questions run over every state through :meth:predict_batch (grouped forward passes, results in input order), then each state's answers are projected as decide does. Extra keyword arguments (batch_size=, model=, hooks=, ...) are forwarded to predict_batch. See laya.structured.

route_batch ¶

route_batch(
    requests: Sequence[Dict[str, Any]],
    hooks_timeout: Optional[float] = None,
    *,
    hooks=None,
    hooks_raise: Optional[bool] = None,
) -> List[RouteDecision]

Route a heterogeneous request batch without loading any checkpoints.

Each request is a mapping with state and questions plus the same optional routing overrides accepted by :meth:route: model, task, lang and lang_guess. The returned decisions preserve input order.

This is intentionally separate from inference so callers can inspect or aggregate routing decisions before paying model-load cost.

Parameters:

requests (Sequence[Dict[str, Any]]) –

Sequence of request dictionaries, each requiring state and questions.
hooks_timeout (Optional[float], default: None ) –

Override the Router's hooks_timeout for this call, applied to every request's on_route dispatch, as on :meth:route.
hooks (HookArg, default: None ) –

Per-call hook or sequence of hooks for this call.
hooks_raise (Optional[bool], default: None ) –

Override the Router's hooks_raise policy for this call.

predict_batch ¶

predict_batch(
    requests: Sequence[Dict[str, Any]],
    batch_size: Optional[int] = None,
    hooks_timeout: Optional[float] = None,
    min_confidence: Optional[float] = None,
    sort_by_length: bool = False,
    *,
    hooks=None,
    on_predict_start=None,
    on_predict_end=None,
    hooks_raise: Optional[bool] = None,
    diagnostics: Optional[Dict[str, Any]] = None,
) -> List[Dict[str, Any]]

Route and execute a heterogeneous request batch with minimal model churn.

Requests are routed first and grouped by checkpoint. Within each checkpoint, requests that share the same question schema are passed to Agent.predict_batch so their states can share forward passes. Results are then restored to the original request order.

Requests may independently specify model, task, lang, lang_guess, max_len or head_max_len and may use different question schemas. max_len / head_max_len are the per-request form of the token-budget override predict takes as call arguments: they set the checkpoint's state and question-head budgets for that one request, so a wide question can be asked without shrinking the batch's other requests to the same window. Requests that ask for different budgets are split into separate forward passes, since one Agent.predict_batch call carries one budget for all its states. A start hook may still replace either value on ctx.

Router-level predict hooks run per request, as predict runs them: each request gets its own PredictContext, so on_predict_start can replace that request's state, questions or token budget, or ctx.skip(...) it, and on_predict_end sees and may replace its result. Requests are grouped for the forward pass after their start hooks have run, and a checkpoint group's requests end in reverse of the order they started. If a checkpoint group fails, every request of it whose start hook ran fails with the exception, a cache hit included: each gets on_error and then on_predict_end before the exception propagates.

Parameters:

requests (Sequence[Dict[str, Any]]) –

Sequence of request dictionaries. Every item requires state and questions and may include model, task, lang or lang_guess routing overrides and max_len / head_max_len token-budget overrides.
batch_size (Optional[int], default: None ) –

Optional maximum number of states per Agent forward-pass batch.
hooks_timeout (Optional[float], default: None ) –

Override the Router's hooks_timeout for this call.
min_confidence (Optional[float], default: None ) –

Optional float or per-bucket mapping for confidence gating.
sort_by_length (bool, default: False ) –

Forwarded to every Agent.predict_batch call, so each question group pads to a shorter maximum; see Agent.predict_batch. Results retain the input order either way. Silently dropped for an attached agent whose predict_batch predates the knob (#294).
hooks (HookArg, default: None ) –

Per-call hook or sequence of hooks for this call.
on_predict_start (PredictHookArg, default: None ) –

Plain callable or sequence of callables for start events.
on_predict_end (PredictHookArg, default: None ) –

Plain callable or sequence of callables for end events.
hooks_raise (Optional[bool], default: None ) –

Override the Router's hooks_raise policy for this call.
diagnostics (Optional[Dict[str, Any]], default: None ) –

Optional dict that receives the per-group sort outcome under the key "sort_by_length" once the batch has run: how many question groups actually reached an agent, how many of those could reorder, and the smallest / largest group size. The reorder switches per group inside this call, so a caller that wants to say whether the knob did anything (#1111) reads this instead of guessing from the request count beforehand. Untouched when sort_by_length is false or no group reached an agent.

Returns:

List[Dict[str, Any]] –

One normal Router prediction result per request, in the same order as the input.

RouteDecision ¶

Bases: dict

The routing outcome: which model, why, and what was detected.

Behaves as a dict so it serialises straight into an API response.

DEFAULT_MODELS `module-attribute` ¶

DEFAULT_MODELS = {
    "english": (BUNDLE_REPO, None),
    "multilingual": (BUNDLE_REPO, "multilingual"),
    "typed-decisions": (BUNDLE_REPO, "typed-decisions"),
}

Router¶

Router ¶

registered property ¶

loaded_revisions property ¶

resolve ¶

register ¶

unregister ¶

load ¶

attach ¶

preload ¶

unload ¶

route ¶

predict ¶

predict_long ¶

decide ¶

decide_batch ¶

route_batch ¶

predict_batch ¶

RouteDecision ¶

DEFAULT_MODELS module-attribute ¶

registered `property` ¶

loaded_revisions `property` ¶

DEFAULT_MODELS `module-attribute` ¶