BlockReuseConfig#

class tensorrt_llm.llmapi.BlockReuseConfig( *, policy: Literal['all_reusable', 'per_request', 'per_conversation'] = 'all_reusable', max_num_turns: Annotated[int, Gt(gt=0)] = 1, swa_endpoint_rewind_tokens: Annotated[int, Ge(ge=0)] = 0, enable_branch_snapshot: bool = False, )[source]#

Bases: StrictBaseModel

Configuration for KV cache block reuse policies.

field enable_branch_snapshot: bool = False#: Keep reusable state where a request’s prompt diverges from the prefix cache, so later requests sharing that prefix can reuse up to the fork. For sliding-window attention (SWA), the SWA window at the fork is committed. This helps when only the SWA window at a chunk end is kept: policy ‘per_request’ or ‘per_conversation’, or SWA scratch reuse. The fork is aligned down to a block boundary and becomes a forced context chunk boundary. SWA layers skip this with joint target/draft KV cache reuse. For hybrid Mamba models, the recurrent state is snapshotted at the fork. If no periodic or additional snapshot point applies to the prompt, the prompt end is snapshotted as a fallback. Requires KV cache manager v2 and block reuse.

field max_num_turns: Annotated[int, Gt(gt=0)] = 1#

Maximum number of completed conversation turns whose committed SWA-window blocks and Mamba stable-boundary state are retained by KV cache manager v2. Only used when policy is ‘per_conversation’.

Constraints:

gt = 0

field policy: Literal['all_reusable', 'per_request', 'per_conversation'] = 'all_reusable'#: KV cache manager v2 block reuse policy. ‘all_reusable’ commits reusable blocks after every context chunk; ‘per_request’ commits them only after the final context chunk; ‘per_conversation’ uses ‘per_request’ commits and retains committed SWA-window blocks and Mamba stable-boundary state for up to max_num_turns completed turns. Periodic Mamba state snapshots are disabled with ‘per_conversation’. All reusable blocks remain subject to normal cache eviction. Requests without conversation params use ‘per_request’ behavior. When ‘all_reusable’ and SWA scratch reuse are both enabled, only non-scratch blocks are committed for reuse.

field swa_endpoint_rewind_tokens: Annotated[int, Ge(ge=0)] = 0#

Extra tokens before the final SWA window whose cache blocks receive higher eviction priority, together with sink blocks. Zero disables the entire endpoint-priority callback. Positive values require KV cache manager v2, block reuse enabled, and policy=’all_reusable’. This preference does not guarantee residency or change attention windows or prefix matching. Dummy and draft requests are excluded.

Constraints:

ge = 0

__init__(**data: Any) → None#

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

validator validate_swa_endpoint_policy » all fields[source]#