Helpers¶

Full detection result for a state.

Returns script, script_profile, language (best effort, may be None), is_english, non_latin_fraction and mixed_segment. mixed_segment names the line or field the segment scan itself singled out, else None: a state flipped non-English by the per-value read below is reported with mixed_segment still None, because that read keeps the offending value in its own local and never writes it back.

String values are what get read. When a state has several of them, one non-English value is enough: joining every value into one window let a long English note fill the 4000 characters, or outvote a short German message, and that message was then sent to the English checkpoint (#384). The segment scan still stops at 4000 characters, which is what keeps a huge field cheap; a value it did not reach is read on its own afterwards.

Dominant script of text: 'latin', 'han', 'devanagari', ... or 'unknown' if there are no letters.

True when the English checkpoint can be expected to read this state.

Remove quoted email history, signatures and disclaimers to keep input focused.

max_chars is the length the result is cut to, 3000 characters unless raised -- see email_state, which takes the same budget and passes it through.

It is a slice budget, not a floor, and it is applied twice: as the max_chars * 4 pre-cut that bounds the regex work below, and as the text[:max_chars] that returns the result. The pre-cut runs first, so the quote, signature and disclaimer rules see only the first 4 * max_chars characters -- a footer that begins past that window is never recognized, and the paragraph holding it survives to the final cut. So 0 answers with the empty string, and a negative value trims from the end: its length test is against a negative number, so the pre-cut always fires and removes 4 * |max_chars|, and the final slice removes |max_chars| more, which is 5 * |max_chars| off the body and empty once that reaches the body's length. That is specified behaviour, recorded by the budget-zero and budget-negative-* cases of laya-java/fixtures/email.json that the Java and TypeScript ports are tested against, so "no limit" means a budget larger than the body -- never 0 and never negative.

Construct a clean state dictionary for email classification.

max_chars is the budget clean_email_body cuts the body to, and it is worth raising for a long message: at the default the body stops after 3000 characters, so a request that arrives in the last paragraphs never reaches the model -- including through predict_long, which scans a state in windows precisely so it can read past one window's worth. Lowering it does the same thing more surely: it is a slice budget, so 0 gives the model a subject and an empty body, and a negative value cuts from the end, which is where that late request lives. clean_email_body states the arithmetic, and the budget-zero and budget-negative-* cases of laya-java/fixtures/email.json pin it for every port, so it is behaviour rather than an error -- pass a budget larger than the body when the message is long, never 0 or a negative to mean "no limit". Ignored when clean=False, which passes the body through whole.

Any other keyword becomes a field of the state, so it is read by the model; a typo here is an input mutation, not an error. The exception is a keyword whose value is exactly None: it is dropped rather than added, so email_state(..., thread_id=None) gives the same state as leaving the keyword out. The key is absent, not null -- and the distinction reaches the model, because laya.common.serialize_state turns the state into JSON before it is tokenized, so a retained None would put "thread_id": null in the prompt. sender is dropped on the same call by a different rule, falsiness rather than None, so sender="" leaves no from field either.

Preset questions for customer support ticket triage.

Preset questions for inbound email triage and threat filtering.

Preset questions for real-time LLM input guardrails.

Preset questions for content safety and moderation.

Preset questions for intelligent model routing.

How many state tokens q leaves inside max_len, which is what build_sequence keeps.

The head is the question's own -- its instructions plus one [MASK]-prefixed span per option -- so a question with many options leaves less room for the state than one with two, and two questions in the same request do not have to leave the same amount. Anything past the return value is cut off (the start is kept, or the end for a conversation turn list).

Window size and stride for scanning a state that is longer than one sequence.

predict_long decodes each token window back to text and scores it as an ordinary state, so a window wider than the room the questions leave is re-truncated by build_sequence on the way in: the tail of every window reaches no model, while the reported span says it did. The window is therefore capped at state_room, and at the smallest room of questions, because the windows are one list of states scored for every question in shared forward passes -- a window sized for the roomiest question would be cut short for the tightest one, and the offsets reported on its answers would mean something different per question. With no questions there is nothing to fit, so the caller's window (or the checkpoint default) stands. That default is max(64, max_len - head_max_len - 8): the state budget the config leaves, with a 64-token floor under it, so a widened head_max_len stops shrinking it there.

questions are internal question dicts, as Agent._to_internal returns them.

An explicit window wider than the room is clamped to it with a RuntimeWarning, since a scan at the requested size cannot read what it claims to. A stride past the effective window is refused: the tokens between two windows would be read by neither, which is the failure predict_long exists to prevent. The default stride keeps its 50% overlap of the effective window, so a span near a boundary still lands whole inside some window; an explicit stride equal to the window still reads every token, with no overlap to catch a span that straddles a boundary.

Returns (window, stride, room).

The questions whose options no longer have a token span each, from per-item stats.

total is the number of options the question defines, not the number of markers that reached the sequence: a report counted from the markers would say "43/58" about a request where 28 options never made it into the input at all.

Return the top-k choice labels for state.

embed_fn maps a list of strings to an array of shape (len(texts), dim). It is called once, with the query text first and then one string per option in criteria order. Option strings match render_options for a choice question.

When k is at least the number of labels, every label is returned in its original order and embed_fn is not called.

Ties keep the earlier label. Ranking is a signed cosine, not a similarity floor: a label that scores 0 -- no signal at all, or a non-finite vector treated as one -- does outrank an earlier label that scored negative, and k drops the negative labels first.

With return_scores=True the return is the (labels, scores) pair, where scores holds the signed cosine per kept label in rank order -- the same values predict_shortlist reports in its shortlist metadata. scores is None when nothing was dropped, exactly as in that metadata.

Shortlist each choice question, then call predict or system_one once.

Non-choice questions are forwarded unchanged. A choice whose label count is <= k is forwarded unchanged and does not call embed_fn. The caller's questions dict is not mutated.

A narrowed choice that carried an option_order has it dropped, with one RuntimeWarning naming the question: the order has one slot per option, so an order that fitted the caller's label set no longer fits the kept labels and the agent refuses the question. Remapping the order onto the kept labels would silently move options between slots, so it is dropped instead. A passthrough choice and non-choice questions keep theirs.

The returned dict is the model result plus a shortlist entry. Probabilities on a shortlisted choice are over the kept labels only. shortlist has an entry for each choice question; a non-choice id is absent, because no ranking ran for it. Each entry holds labels, scores, k, n, and passthrough. labels is the rank order a shortlist produced, or the criteria order itself when passthrough is set and no ranking ran, and it always holds min(k, n) labels; scores is the signed cosine of each kept label in that order -- negative included, never clamped to 0 -- or None when nothing was dropped. Neither k nor n counts what labels kept: k is the cap the caller asked for, which a question with fewer labels than that does not reach, and n is the caller's label count before any narrowing.

Extra keyword arguments are forwarded to predict / system_one (for example model= on a Router).

Narrow each large choice question by elimination, then call predict once more.

A choice with more than group_size labels is cut, in criteria order, into groups of near-equal size and at most group_size. One predict call answers every group of every such question -- the groups go in as separate questions, so they share one forward pass -- and each group's answer goes through to the next round. Rounds repeat until no choice has more than group_size labels left; at the default 16, up to 256 labels take one round. Unlike predict_shortlist this needs no embedder, and each label is read with the option budget of a group_size-label question, not of the whole label set.

The final call answers every question of the request, with each choice that went through a round cut to its finalists. Non-choice questions and choices of at most group_size labels go to it unchanged, so when nothing needs a round it is the only call. The caller's questions dict is not mutated.

A choice that went through a round loses an option_order it carried, with one RuntimeWarning naming the question: the group and finalist questions hold fewer labels than the caller's, and an order no longer fits the kept labels. Remapping it onto them would silently move options between slots, so it is dropped instead. A choice that needed no round keeps its order.

The returned dict is the final call's result plus a tournament entry. It has an entry for each choice question -- a non-choice id is absent, since such a question plays no round -- holding labels (the finalists, in criteria order), n and rounds. n is the caller's label count before the first round, the same meaning as in predict_shortlist's metadata: it says how wide the field was, not how many finalists came out of it, so on a choice that played a round it is larger than len(labels). Probabilities, confidences and usage come from the final call, so a tournament choice's probabilities are over its finalists only.

Every call's reply is checked against the same shape, the elimination rounds included: a reply that is not a dict, and one whose answers is not a dict, each raise that contract by name instead of an indexing error or a bare KeyError from inside the module. What an answer holds is not checked: the final call's answers go to the caller as the agent wrote them.

Extra keyword arguments are forwarded to every call (for example model= on a Router).

Mean-pool the checkpoint encoder already loaded on agent.

The callable embeds a list of strings with agent.tok and agent.model.encoder. It does not run the decision head and does not download weights. A dedicated bi-encoder passed as embed_fn will usually shortlist better; this helper is for callers who only have the Laya checkpoint in memory.

Padding positions are excluded from the mean. The encoder's train/eval flag is left as the caller set it (a loaded Agent is already in eval). Each call uses the current agent.device, including after CPU fallback.

Cache embed_fn output per input string, under an LRU bound.

predict_shortlist embeds the query plus every option text on each call. When the same option set is shortlisted on every request -- a fixed intent or label list, as in the README's BANKING77 example -- the option rows do not change between calls, yet they are re-embedded every time. Wrapping the embedder once::

embed_fn = cached_embed_fn(embed_fn_from_agent(agent))

leaves the first call's embedder work unchanged and reduces each repeat call to embedding the new query alone.

Lookups are exact string matches. Texts missing from the cache are deduplicated and embedded in a single embed_fn call, so a cold cache costs the same number of batched calls as the unwrapped function. Rows are stored as float32; the cache holds at most maxsize strings and then evicts the least recently used entry, bounding memory at about maxsize * dim * 4 bytes. Nothing is cached when embed_fn raises or returns a bad shape. A non-finite value is not a bad shape: NaN, inf and -inf become 0.0 on the way in, the same replacement :func:_embeddings makes for an unwrapped embedder, and the repaired row is then cached. One transient NaN in an option's embedding therefore ranks that option as a zero vector on every later request until cache_clear(), while the unwrapped embedder would simply re-embed it -- shortlist membership can differ between the two on the second call. The float32 storage shows in the scores too: a wrapped first call can differ from the unwrapped one in its last digits.

The wrapper is safe to share between threads: the lock covers only cache reads and writes, never the embedding call. The returned callable carries cache_info() -- a dict with size, maxsize, hits and misses -- and cache_clear(). Clear the cache if the model or weights behind embed_fn change.

Validate opt-in abstention threshold min_confidence (#361, #394).

Either a real number in [0.0, 1.0] (one threshold for every answer; booleans rejected even though isinstance(True, int)), or a per-bucket mapping (see :func:check_min_confidence_map) so the threshold can differ by option count. Returns the value in its validated form -- a float for the scalar case, a dict[str, float] for the mapping case -- which the gate functions below both accept.

Validate a per-bucket abstention-threshold map (#394).

Keys are the option-count buckets in common.temp_bucket's spelling -- every :func:bucket_keys member ("choice:2", "choice:3-5", "score:6-10", "noul:2" and so on) -- plus the optional "default" used for any bucket the map does not name. Anything else raises: a key no answer can produce is not an unused entry, it is a bucket the caller believes is gated and is not, because :func:resolve_min_confidence falls past it to "default" and then to 0.0 -- gate nothing. Values are floats in [0.0, 1.0]. One confidence threshold does not transfer across option counts (#394); this lets a caller gate each bucket at the level its calibration actually earns. Fit one with :func:laya.calibrate.fit_abstention_thresholds, whose keys are this vocabulary by construction.

The threshold this answer's option-count bucket is gated at, under a per-bucket map.

Falls back to the map's "default" entry, then to default (0.0 -- gate nothing), for a bucket the map does not name, so an unconfigured bucket never abstains by surprise. A map that went through :func:check_min_confidence_map can only miss here by omitting a bucket: a key mis-spelled well enough that no answer produces it is refused at validation, because it would otherwise reach this fall-through and leave that bucket gated at nothing while its caller believes it is gated.

Opt-in abstention marker (#361): flag answers whose confidence falls below min_confidence.

Reads answer_confidence (max(p), the quantity the calibration figures describe -- the level it reaches still moves with the option count, which is why min_confidence takes a per-bucket map), falling back to confidence if answer_confidence is absent. The raw answer and confidence stay intact; low_confidence: True is added when the answer falls below the threshold, and removed if a previously-flagged answer now clears it (e.g. when a result dict is reused or re-evaluated with a different threshold).

min_confidence is either a float (one threshold for every answer) or a per-bucket mapping (#394), in which case each answer is gated at the threshold of its own option-count bucket via :func:resolve_min_confidence.

Exactly 0.0 runs this same loop rather than short-circuiting out of it. Nothing falls below it, so it marks no answer -- and the pass is what clears a marker an earlier, stricter threshold left on a reused payload, which is the state #910 was filed about.

Report the confidence gate's state, on the answers a gate was actually applied to.

A gate is a policy, and a policy whose application cannot be observed is not one. With min_confidence set, this writes abstention -- one of :data:GATE_STATES -- onto every answer, plus abstention_threshold, so a caller can answer three questions it otherwise cannot:

• what fraction of decisions abstained, rather than inferring it from whether low_confidence happened to be set;
• how many answers the gate could not decide on, which a boolean cannot express at all;
• what threshold produced these results -- flag_low_confidence consumes the threshold and drops it, so without this a batch run with per-class thresholds cannot be re-split.

GATE_UNEVALUATED is the case a boolean cannot express: the gate ran and the answer carried no usable confidence, so the gate could not decide. Reporting that as a pass is the same lie as reporting it as a flag.

With min_confidence unset, this writes nothing. No abstention, no abstention_threshold, no flag. That is the whole contract: an ungated call returns exactly the payload it returned before, and the presence of the field -- not a fourth value read out of it -- is what tells a caller the gate ran. Call it unconditionally, once per call, in place of an if min_confidence is not None: guard: that guard is what leaves a path reporting nothing at all, which is the state this function exists to distinguish.

The flag itself stays :func:flag_low_confidence's -- this delegates rather than re-implementing the rule, so the boolean and the reported state cannot drift apart.

A min_confidence of exactly 0.0 was set, so states are reported. Nothing can fall below it, so :func:flag_low_confidence marks no answer at it and clears whatever a stricter threshold left on a reused payload. Every answer carrying a usable confidence therefore reads passed, and the threshold echo is what distinguishes that from a real pass at a real threshold.

Probability mass on the answer being reported: max(p).

This is the quantity temperature scaling fits, and the quantity every calibration figure in this repository is computed on -- both benchmark harnesses take conf = max(probs) before calling ece_score. The README's gating section relies on the property that goes with it: of the answers returned at confidence c, about c of them are right. That property is conditional, and the condition is not met by default -- it holds only after the temperatures have been fitted and validated on held-out data for this checkpoint and this option count. The shipped checkpoints are over-confident: choice:11+ is fitted at 0.1006, a ~10x sharpener that would publish a genuine 0.24 top probability as 0.9999. clamp_temperature refuses it and applies TEMP_MIN (0.5) instead, which still sharpens 2x and publishes that same 0.24 as 0.4993 -- so a threshold applied to these numbers still selects below model accuracy (issue #394). The clamp bounds the damage; it does not calibrate anything.

confidence_from_probs below reports a different quantity on a different scale and carries no such guarantee, so the two must not be compared against the same threshold.

Normalized Shannon entropy confidence: 1 - H(p) / log(k).

How concentrated the whole distribution is. Useful, but not calibrated: it is not what temperature scaling fits and not what the reported ECE measures. See answer_confidence.

The formula is reported only for k >= 2: log(1) and log(0) are zero, so a distribution over fewer than two options has no entropy scale to divide by. Those inputs return 1.0, the maximum score this function can produce, whatever p holds -- an empty or single-option question reads as perfectly concentrated, and a caller comparing the reported confidence against any threshold sees a pass. That is not the same as the abstention gate passing it: laya.confidence gates on answer_confidence first and reaches this number only when that field is absent. The gate does pass a real one-option answer, for a different reason -- a softmax over one logit is exactly [1.0], so both numbers are 1.0. laya.structured builds such an answer from {"enum": ["solo"]}, and the option count is bounded only above (MAX_OPTIONS).

The confidence Jev would report for this answer's probabilities, or None (#302).

Laya's confidence is normalized entropy, and Jev's is a different formula, so a threshold carried over from Jev selects about twice as much on Laya. These are TypeSafe's own formulas (system-one-adapter-python, _utils/confidence_metrics.py):

choice  (max(p) - 1/n) / (1 - 1/n)
score   max(0, 1 - sum(p_i * |i - mode|) / D), D the same sum for a uniform distribution

Both are 1.0 for a single option, and a zero total is read as uniform, as the adapter does. A noul answer has no Jev confidence, and a malformed document -- a probability that does not read as a number, a boolean, a non-finite or a negative value -- gives None rather than a guess, so a caller filtering on this number never filters on a made-up one. Negatives are the case that needs naming: the division below normalises by the sum, so a mixed-sign document can divide by a near-zero or negative total and return a number outside [0, 1] for a distribution whose argmax is not the one it measured, while an all-negative document divides back into range and returns exactly what its positive counterpart returns. Refusing them is what makes the range a property: every number this returns lies in [0, 1].

Expected Calibration Error across confidence bins.

bins is the number of histogram buckets and must be an integer of at least 1. It is refused rather than misread because np.linspace(0, 1, bins + 1) yields no interior edge for 0 and -1: the loop body never runs and the function returns 0.0, the best score ECE can report, for a distribution that is anything but well calibrated. A count that arrives from a config file as 0 therefore turns a calibration gate into one that always passes. True is refused with the same argument -- it would silently histogram into a single bucket. Any other Integral is accepted, a numpy.int64 among them, since the arrays here are numpy's own.

Fit one scalar T by NLL + LBFGS on log T.

The result is clamp_temperature of the optimised scale, so it lies in [TEMP_MIN, TEMP_MAX] (or is the neutral 1.0 when the value is not a number). The search is bounded to LOG_T_MARGIN e-folds outside that band, which covers any minimum the answer can use; past it the float32 scale drifts toward exp() underflow, where the objective divides by zero and every gradient becomes NaN, and clamp_temperature reads that NaN back as the neutral 1.0. Returns 1.0 when fewer than min_n pairs are given. min_n defaults to MIN_BUCKET_N (the per-bucket floor). Type-level fits pass MIN_TYPE_N, which is lower, so a dataset that fills no bucket still gets a scalar instead of staying at 1.0.

Fit type-level scalars and per-bucket temperatures.

MIN_BUCKET_N is the per-bucket floor: smaller buckets are omitted from temperature_by_options and the type-level scalar covers them. MIN_TYPE_N is the separate, lower floor for that scalar only.

compute_ece=False (the default, and the path Agent.fit_temperatures stores) fits on every record and returns no report key. seed is ignored on this path.

compute_ece=True holds out ECE_HOLDOUT_FRAC of each bucket, stratified by temp_bucket, using seed so the same records always split the same way. Temperatures are fit on the remainder only and ECE is scored only on the held-out records. report["n"] is the number of records passed in; report["n_eval"] is the held-out count the ECE rests on. A bucket that would drop below MIN_BUCKET_N after the holdout is fit on all of its records, left out of the eval set, and named in report["buckets_excluded_from_eval"] instead of being dropped. n_by_bucket always counts the full input, including when the fit itself used a subset.

Fit a per-temp_bucket abstention threshold so a gate keeps a target error in each fitted bucket.

A single min_confidence does not transfer across option counts (#394): the calibrated confidence of a 2-option and a 12-option answer live on different scales, so one cut over- or under-abstains depending on the question. This fits one cut per bucket instead, keyed exactly like temperature_by_options (common.temp_bucket, e.g. "choice:3-5"), and the result is a min_confidence map that :func:laya.confidence.check_min_confidence / :func:laya.confidence.apply_confidence_gate accept directly -- with one exception to keep: every bucket below min_bucket_n is omitted, so a calibration set too small to fill a single bucket returns {}, and the validator refuses an empty map ("must be a non-empty dict"). Pass no gate in that case rather than passing what came back; laya.train writes the map into a checkpoint only if abstention_thresholds:.

records are the same (qtype, logits, target[, k]) tuples fit_temperature_map consumes (records_from_labeled builds them). Confidence is the calibrated max(p) -- the logits are scaled by the fitted temperature / temperature_by_options first, so thresholds and the numbers the runtime reports are on the same scale. target_error is the tolerated error among accepted answers; min_bucket_n omits buckets too small to fit, and conservative adds a one-sample margin. An omitted bucket is not caught by a fallback: the map lookup falls to its own "default" entry and then to 0.0, which gates nothing, so a min_bucket_n that excludes a bucket leaves that bucket ungated unless the map names a "default". The thresholds are empirical cuts on the calibration set, not a formal coverage guarantee -- validate on held-out data (fit_temperature_map(..., compute_ece=True) gives a held-out split) for a production gate.

Pass binning_map when the agent that will serve these thresholds has one installed -- by Agent.fit_binning, or by a calibration payload that carries binning_map -- because the runtime recalibrates answer_confidence through that map before anything reads it, so a cut fitted without it is a cut on a scale the gate never sees. The thresholds are then on the binned scale, and the order the two were fitted in stops mattering. Measured on 1,200 synthetic 12-option records at target_error=0.10: the cut fitted without a map holds 9.8% error over 50% coverage on un-binned confidences, and admits 94.5% of answers at 25.6% error once the same number is compared against binned ones.

Fit a per-temp_bucket histogram-binning recalibration map for answer_confidence.

Temperature scaling applies one scalar per bucket; it cannot fix a bucket whose reliability curve is not a simple sharpening/softening (the pathological choice:11+ the shipped English checkpoint carries is one). Histogram binning is the non-parametric alternative: split the calibrated confidences of a bucket into bins equal-width bins over [0, 1], and map every confidence that lands in a bin to that bin's empirical accuracy. It needs no monotonicity assumption and no extra dependency (NumPy only; isotonic regression would pull in scikit-learn).

records are the same (qtype, logits, target[, k]) tuples fit_temperature_map consumes; confidence is the calibrated max(p) (logits scaled by the fitted temperature / temperature_by_options first), so a binning map composes on top of a temperature map rather than replacing it. Returns {bucket: {"bins": N, "values": [recalibrated confidence per bin]}}; buckets below min_bucket_n are omitted. Apply it with :func:apply_binning_map. An empty bin (a confidence range the calibration set never produced) maps to that bin's centre, a point inside the region the value came from: the value is still rewritten, and moves by at most half a bin width. What the centre buys is that an unseen confidence is never recalibrated to a fabricated 0, not that it passes through -- only a bucket the map has no entry for does that (:func:apply_binning_map).

Recalibrate one answer_confidence for its option-count bucket (common.temp_bucket).

Returns the confidence unchanged when the map has no entry for the bucket, so a bucket the map was not fit for passes through rather than being forced to a wrong value.

Fit a histogram-binning map on top of this agent's fitted temperatures and store it.

records are the same (qtype, logits, target[, k]) tuples as fit_temperatures consumed. The map is keyed exactly like temperature_by_options, composes on top of the current temperatures, and is written out by save_calibration as binning_map.

Render option texts in label-index order. Noul semantic order is always [false, true].

Strictly proper scoring rule reward: log score + spherical score + ranked probability score.

q: [..., N, K] reported distributions target: [N, K] (one-hot or soft target distributions)

TD(lambda) targets for multi-turn conversation trajectories.