Python package version

The Python wheels for this release are published as 1.4.0rc2.post1. Container images and documentation remain at 1.4.0rc2. Use 1.4.0rc2.post1 when installing the Python package.

The wheel-only patch removes a direct Git dependency that prevented PyPI publication (#21128) and updates the Boost download mirror used during builds (#21133). Patched wheel source: v1.4.0rc2.post1, commit 43f8f70da6415a3b06f91f9ebaa4fb7611ab62cc.

Highlights

• Known Issues

  • Tensor-parallel serving with RPC orchestration may hang during model initialization when reusing an MPI worker pool across threads.

• Model Support

  • Add Cosmos3-Edge Reasoner support for visual generation workloads #20307
  • Add GLM-5.2 NVFP4 low-latency decode on Rubin through Modeling V2 #20517
  • Enable MiniMax-M3 on Rubin with NVFP4 KV exemptions and paged FMHA #20167
  • Expand Qwen3.8-Flash-Next Modeling V2 support for MTP, video, and quantized KV caches #20276
  • Add SM103 Modeling V2 defaults for GPT-OSS-120B and DeepSeek-R1-0528 NVFP4 #20267
  • Qualify dense Phi3 models for ModelExpress integration #19997
  • Support static FP8 Cosmos3 parallelism and four-step distilled checkpoints #20007
  • Support FLUX.1-dev static NVFP4 with per-tensor FP8 attention on SM103 #18396

• API

  • Migrate LlmRequest fields from C++ to Python (BREAKING) #20388
  • Drop KV cache manager V1 support from transceiver V2 (BREAKING) #20129
  • Remove obsolete executor backend compatibility paths (BREAKING) #20043
  • Enable zero-copy mode for Mooncake Store (BREAKING) #20108
  • Deprecate n-gram and user-provided speculative decoding modes #20438
  • Add configurable FP32 LM-head logits through lm_head_output_dtype #19850
  • Consolidate prompt rendering into standalone serve/render support for Dynamo #19964
  • Define backend contracts and share lifecycle enforcement for KV cache transfers #20117, #19822
  • Add a blob-store driver boundary with an in-memory test implementation #20217

• Feature

  • Add native retention and cross-turn generation-block reuse to KV cache V2 #20086, #19710
  • Commit sliding-window prefixes at fork points and account for connector-provided reuse blocks #19877, #20180
  • Add first-N/last-N mixed-precision KV caching using native sliding-window attention #20452
  • Support FP4 MLA in disaggregated serving deployments #20544
  • Add MXFP8 refitting with piecewise CUDA graphs and Ray collectives #20075
  • Optimize DeepSeek-V4 and DSpark on Rubin, including fused RMSNorm/RoPE kernels #19961, #20366
  • Add Kimi K3 fabric gathering, TCPStore rendezvous, and managed KDA replay buffers #20576
  • Integrate FlashInfer KDA prefill with fused convolution #20653
  • Enable shared CUDA graph setup for Qwen3-family vision-language encoders #20213
  • Schedule multimodal encoders by prefill chunk and reduce Gemma attention-mask memory #19884, #20405
  • Improve Qwen-Image Ulysses performance and add MiniMax-H3 asynchronous Ulysses execution #17147, #20333
  • Add PrimsTS SageAttention for VisualGen VSA and SOL workloads #20224
  • Parallelize MiniMax-M3 indexer scoring and use BF16 tensor-core router GEMMs #19975, #20699
  • Accelerate GLM-5.3-Flash sparse attention, verification offsets, mHC, quantization, and routing #20454, #20389, #20390
  • Improve DSA paged-MQA logits kernels and default indexing to CuTe DSL #19976
  • Optimize one-model speculative decoding host scheduling, target logits, and KDA draft records #19989
  • Optimize GVR V2 for large batches and short sequences #20106
  • Fuse SwiGLU into BF16 gate/up GEMM epilogues on SM100 #19773
  • Compile Mamba replay-input preparation and vectorize SID-GR final beam assembly #20688, #20166
  • Support required tool choice for Nemotron Super 3.5 VL #20422

• Fix

  • Prevent global stop-word buffer growth in the PyTorch sampler #20679
  • Preserve original errors when lazily importing serving modules and submodules #20948, #20660
  • Retry wrapped connection resets within the transient retry budget #20889
  • Respect Gemma4 encoder budgets and fix startup on non-SM100 GPUs #20663, #18547
  • Preserve admitted LoRA adapter formats and restrict multimodal adapters to decoders #20774, #20605
  • Bound LoRA parameter-fill reads to valid module output sizes #20477
  • Correct MiniMax-M3 rendered prompts, video counts, multimodal speculation, and KV reuse #20626
  • Fall back to tokenizer chat templates when multimodal processors provide none #20880
  • Update live-token metadata before constructing sparse-attention fields #19721
  • Apply NVFP4 AWQ pre-scales and expert input scales in MoE loading #20823, #20547
  • Preserve MXFP8 fused weight-scale alignment when loading weight rows #20443
  • Support MegaMoE CuTe DSL TEP across models and CUTLASS MXFP8 on SM107 #20055, #19953
  • Restore CUTLASS MoE compatibility, feature checks, and INT8 tensor-parallel padding #19753, #16198
  • Correct CuTe DSL input-quantization synchronization for prefill and multi-stream execution #20779, #20808
  • Keep low-M GEMM outputs distinct during CUDA graph capture #20334
  • Honor separate value strides and support FP8 custom-mask context attention #20242, #20387
  • Stabilize deterministic GDN decode recurrence and pad DeepSeek-V4 compressed-FP8 scratch #20226, #20119
  • Retain GLM-5.3-Flash MTP draft layers in the target KV cache #20742
  • Recover executor loops after failed forwards and avoid leader-precheck deadlocks #19465, #20272
  • Keep serving metrics responsive and preserve MPI shutdown requests during teardown #20698, #20608
  • Resolve attention-DP dummy roles and flush overlap KV events on every rank #20325, #20314
  • Pass image inputs through the Responses API to multimodal models #20231
  • Use consistent video hashing and avoid GIL-held copies during VisualGen encoding #19920, #20575
  • Clean up cancelled video encodes and remove TRTLLM_VIDEO_ASYNC_ENCODE #19401
  • Resolve Wan quantization per transformer and handle per-token I2V TeaCache timesteps #20413, #20594
  • Upgrade PyNvVideoCodec to version 2.2.0 or later #20738
  • Repair indexer-Q prewarming imports and lazy attention package exports #20734, #20606
  • Use string cache salts with KV cache manager V2 #20048

• Documentation

  • Document LoRA performance tuning and clarify unified TRTLLM_CACHE_DIR behavior #20611, #20607
  • Explain Nemotron tool parsing and correct sliding-window and long-sequence guidance #20525, #20347, #20348
  • Repair stale documentation links and constrain generated Sphinx page exceptions #20598, #20756, #19405
  • Refresh MoE performance dashboard data and clarify post-merge CI descriptions #20798, #20475
  • Remove the issue-approval requirement from contribution guidelines #20630

• Test & Infra

  • Upgrade NIXL to 1.5.0, UCX to 1.23.x, and CuTe DSL to 4.8.0 #20099, #20034
  • Build tritonrelease images from source and unify Docker orchestration and nightly wheel selection #10574, #20596
  • Enforce container wheel version and source provenance; align nightly BOLT behavior #20564, #20612
  • Retain CPU-only unit tests in NGC image sanity checks #20818
  • Add PrimTS vendor promotion tooling and promote the updated vendor source #19048, #20870
  • Synchronize release/1.3 changes back to main #20232
  • Narrow runner interfaces, simplify engine state, and split decoder execution by phase #20270, #20134
  • Remove obsolete warmup paths, runtime switches, and two-model speculative-decoding leftovers #20253, #20464
  • Standardize CUTLASS MoE identities and reorganize domain kernels and component-local tests #20741, #19275, #20658, #20222
  • Expand CBTS coverage, dataclass tracking, docstring analysis, and recorded shadow decisions #20583, #20411, #20410, #20265
  • Classify Slurm failures and preserve post-merge testing after test-list checks fail #19537, #10695
  • Determine changed files from the base commit in merge-queue builds #20456
  • Restore B200 multi-GPU stages after temporary queue relief and shard GB200 coverage #20828, #20899, #20780
  • Add Kimi K3 context-only AgentX and new-model performance-sanity coverage #20613, #20680
  • Add DeepSeek-V4.1 NVFP4 DSpark, Gemma4 assistant, and Qwen3 model coverage #20202, #20671, #20593
  • Restore LTX-2 worker CUDA graph regression coverage #19080
  • Correct forward-error mocks, Qwen token expectations, and KV-cache stored-event assertions #20883, #20673, #20758, #20740
  • Exercise mid-generation KV rebalancing and migrate legacy disaggregated transceiver tests #20479, #19472
  • Use MSA in MiniMax-M3 multi-node evaluation and move hang-detector tests to CPU #20701, #20559
  • Correct performance baseline cutovers, MLA routing thresholds, and benchmark prompt limits #20297, #20652, #20101
  • Report benchmark readiness rejections and repair the performance-triage image reference #20749, #20729
  • Remove DeepSeek-R1 from CI performance-sanity lists #20677
  • Refresh serving configuration schemas and clean up attention-test KV managers on failures #20493, #20522
  • Mirror UCXX downloads, retry wheel builds, and synchronize agent tooling #20080, #19980

What's Changed

• [https://nvbugs/6948011][fix] Shut down attention test KV cache managers on failure paths by @HuiGao-NV in #20522
• [https://nvbugs/6784181][fix] Add an exact-path SPHINX_GENERATED_DOC_PAGES allowlist keyed off helper.py's… by @trtllm-agent in #19405
• [TRTLLM-5432][infra] Fetch UCXX's downloads from the mirror and retry build_wheel.py by @weiminwang-nv in #20080
• [TRTLLM-15766][refactor] Decouple decoder tests from private methods and split DecoderRunner by phase by @lori-ren in #20134
  ...

Highlights

• Known Issues

  • Disaggregated serving with the default Python KV cache transceiver may fail generation requests with a KV cache transfer error after streaming requests are cancelled (observed on A100).
  • The OpenAI JSON-schema serving example (examples/serve/openai_completion_client_json_schema.py) may return an unbounded response with Qwen3 models.

• Model Support

  • Add DeepSeek-V4.1-Flash support #19727
  • Support NVFP4 GLM-5.3-Flash checkpoints #20127
  • Add Qwen3.8-Flash-Next NVFP4 as an experimental Modeling V2 target #20143
  • Support LiLiCorr draft checkpoints in DFlash #19856
  • Remove the automatic W4A16 fallback for Nemotron-H NVFP4 checkpoints on Hopper #15772

• API

  • Default to KV cache manager V2 and the Python KV cache transceiver (BREAKING) #17495
  • Remove unused LlmRequest fields (BREAKING) #19967
  • Replace the C++ ArrayView API with std::span #20205
  • Select Modeling V2 through LLM configuration and arguments #20077
  • Publish serving configuration schemas #19906
  • Add a KV cache lending API for transfer backends #19722
  • Expose native KV event streaming and routing load #19440
  • Support runtime sleep and wakeup through LLM and trtllm-serve, including MiniMax-H3 #19016, #19604
  • Unify compiled-artifact cache configuration through TRTLLM_CACHE_DIR #18897
  • Make KV cache V2 pool rebalancing parameters configurable #19077
  • Reorganize VisualGen attention modules, keeping deprecated import paths and lazy exports #20214, #20215, #20218

• Feature

  • Integrate the Mooncake connector with KV cache manager V2 #19171
  • Fill the KV cache V2 host tier progressively #19650
  • Add RoPE precision options for NVFP4 cold-page compression #19391
  • Honor per-request seeds in batched TorchSampler and one-model speculative acceptance #19719, #20147
  • Add ragged DSpark verification with confidence-based policy and calibrated scoring #19707, #19818, #20071
  • Add deterministic FP32 GVR V2 top-K decoding and reduce its overflow overhead #20100, #20290
  • Add an opt-in setting to compile only piecewise CUDA graph forwards #18366
  • Support Rubin locality-domain execution and extend SM107 and multi-node MegaMoE initialization #19272, #19183
  • Optimize Kimi K3 on Rubin and avoid cuBLAS fallback in KDA prefill #19960, #20084
  • Support packed query heads and compact inputs in Blackwell CuTe DSL MLA #19338
  • Add PrimTS dense BF16, mixed BF16/FP8, and FP8 context attention paths #20006
  • Use paged attention for MiniMax-M3 dense layers and cut NVFP4 KV host overhead #20159, #19974
  • Limit GLM-5.3-Flash indexer scoring to visible pools with BF16 tensor cores #20251
  • Add token-sharded tensor parallelism for Wan in VisualGen #19713
  • Support MiniMax-H3 parallel tiled VAE decoding and Ulysses parallelism #19506
  • Speed up MiniMax-H3 video-VAE decoding and fuse AdaLN RMSNorm modulation #19774, #19772
  • Unify multimodal encoder caching and reduce encoder output copies #17876, #19799
  • Optimize Qwen vision encoders and decode JPEGs through torchvision #19797, #19794
  • Reduce per-response overhead in LLM API streaming #20244
  • Enable SwiGLU clamping and update NVFP4 FC1 weight layout in Rubin CuTe DSL MoE #19488, #20181
  • Overhaul the NVLink one-sided all-to-all #19610

• Fix

  • Align MiniMax-M3 and GLM reasoning and tool parsing between streaming and non-streaming #20442, #19935
  • Flush the Hugging Face streaming detokenizer on the final response #20424
  • Restore trtllm-serve streaming throughput by constraining uvloop below 0.23.0 #19992
  • Constrain FastAPI to versions below 0.140.0 #20161
  • Keep the overlap scheduler responsive while waiting for disaggregated KV receives #19857, #19940
  • Reserve retiring-request slot headroom for overlap scheduling without attention data parallelism #20250
  • Correct draft KV scheduling and prompt-tail recomputation on cache hits with overlap speculation #19745, #19748
  • Size draft-probability buffers by batch capacity and fix speculative beam-search races #19490, #20064
  • Size Blackwell speculative-decoding tree masks for the attention kernel layout #20223
  • Guard Whisper cross-KV reuse when encoder tokens are unavailable #19302
  • Preserve LoRA cache contents when too few pages can be evicted #20212
  • Reclaim idle eager CUTLASS MoE workspace #19981
  • Correct CuTe DSL MoE autotuning inputs and restore FC2 N-tile autotuning #17410, #20277
  • Fix W4A16 NVFP4 MoE dequantization and expert-load-balancing migration, including Nemotron-H #19929
  • Restore routing-weight dtype after NCCL expert-parallel dispatch #20081
  • Enable SM107 NVFP4 CuTe DSL dispatch and fix CUDA-core GEMM for wide FP8 outputs #20204, #20271
  • Enable 96-head MLA in the FMHA autotuner and fix CuTe MLA page-table bounds #18599, #20256
  • Prevent the SM120 DSA KV append crash from a missing stride factor #20062
  • Bound packed indexer prefill logits and use 64-bit offsets for indexer K-cache #20164, #20190
  • Keep DeepSeek-V4 tensor-parallel decode all-reduces off NCCL #20233
  • Avoid deep-copying pending media loads in the DeepSeek-V4 tokenizer #20191
  • Fence Helix all-to-all shared writes before asynchronous bulk copies #20221
  • Match KDA decode fallback layout to prefill #19987
  • Preserve indexed GDN state during Triton autotuning and align recurrent Q/K normalization #19859, #19860
  • Propagate client import paths to remote MPI workers and refresh stale worker environments #19494, #20151
  • Drive MPI progress from a native thread on pipeline-parallel ranks #20030
  • Apply phase-appropriate MoE communication timeouts during warmup #20246
  • Keep single-rank device-mesh topology separate from the job-wide mesh #20021
  • Stop the statistics RPC server when its executor worker shuts down #20158
  • Route FLUX.2 linear_in quantization scales correctly when splitting gate and up weights #20403
  • Fix correctness issues on Rubin and in Kimi K3, KDA, MNNVL, and DFlash paths #19959, #19988
  • Allocate DLPack managed tensors in one block to fix lifetime and shutdown issues #18310
  • Reject paged-context FMHA configurations without the required fused context kernel #19746
  • Honor models that require KV cache manager V2 instead of falling back to V1 #20093
  • Raise a clear error for prompt files with no usable prompts #17733

• Documentation

  • Add Qwen3.5 GB300 agentic-serving guidance and curated configurations #20514
  • Add the MoE Performance Guide and performance dashboard #19546
  • Add a KV cache compression technical blog #20278
  • Document NGC PyTorch wheel installation #20281
  • Describe PyTorch as the sole execution backend and remove stale guidance #19986, #20350
  • Correct the VisualGen constructor in the sparse-attention example #20549

• Test & Infra

  • Upgrade the NGC PyTorch base image to 26.09, retaining the 26.08 forward-compatibility driver #19679, #20252
  • Update dependency constraints for base-image vulnerability fixes #20175
  • Upgrade the Modeling V2 infrastructure #19977
  • Disable BOLT consumption in pre-merge and stop emitting cortex-a53 veneers in BOLT builds #20496, #20024
  • Fix build-system reconfiguration and move C++ tests next to their components #20037, #19886
  • Update trtllm-gen FMHA artifacts and refresh the PrimTS vendor source #20054, #20155
  • Simplify TorchSampler strategy grouping around the fused sampling kernel #20188
  • Add agent-flow orchestration for parallel performance and modeling agents #18876
  • Add GR100 single-GPU post-merge coverage #19894
  • Add Kimi K3 and MiniMax-M3 AgentX performance-sanity coverage and single-phase benchmark lanes #20163
  • Add ModelExpress post-merge accuracy canaries #18561
  • Warm up KV-transfer connections before disaggregated performance-sanity measurements #18433
  • Migrate model, attention, Nemotron, Mamba, Kimi, and disaggregated tests to KV cache manager V2 #20098, #20097, #20046, #20050, #19387
  • Harden disaggregated and multi-node tests with isolated outputs, UCX/NCCL settings, and NVLink-domain placement #20140, #20031, #20189, #19492, #20418
  • Run VisualGen Wan, Qwen-Image, and key-padding tests in CI and split LPIPS goldens by model #20004, #20063, #20198, #19237
  • Run guardwords and release checks in nightly releases and pass source info to PLC scans #20135, #19594, #20449
  • Skip test stages during maintenance, classify CI failures from step logs, and fix rerun JUnit reporting #18353, #19258, #20153

What's Changed

• [None][fix] Refuse paged-context FMHA configurations whose fused context kernel is absent by @brnguyen2 in #19746
• [None][feat] Split VisualGen LPIPS golden media by model family and refresh goldens by @yibinl-nvidia in #19237
• [None][chore] update bench_moe situ support list by @leslie-fang25 in #20009
• [None][fix] Enforce KV correction when overlap scheduling runs with speculation by @brnguyen2 in #19748
• [None][infra] Retry the mypy type-check step in the build stage by @weiminwang-nv in #19589
• [None][infra] Add blossom-ci authorized users by @trtllm-agent in #20025
• [None][fix] Ignore virtualenv and editable-install entries in the pytest sys.path check by @lori-ren in #19872
• [https://nvbugs/6812588][fix] Bound Qwen3.5 batch/seq len in multi-node disagg tests by @reasonsolo in #20019
• [None][fix] GLM tool and reasoning parsers: streaming matches the whole-text parse by @JunyiXu-nv in #19935
• [TRTLLM-10804][infra] Restore release checks in nightly release...

Highlights

• Known Issues

  • Multi-GPU (TP2) streaming or RPC executor workloads may hang during worker initialization when MPI worker pools are reused across LLM instances (observed on H100).
  • DeepSeek MLA models on SM120/SM121 GPUs (for example RTX PRO 6000 Blackwell) may crash with an illegal memory access when CUDA graphs are captured with BF16 BMM.
  • GPT-OSS models on SM120 GPUs may fail with attention data parallelism when automatic communication selects NCCL EP for unsupported hidden sizes.
  • Qwen3 models with tied embeddings (for example Qwen3-0.6B) may fail at model construction when attention data parallelism is enabled with TP greater than 1.
  • OpenTelemetry tracing may export additional FastAPI spans when FastAPI 0.142 or newer is installed.

• Model Support

  • Add GLM-5.3-Flash (glm5_next) support #19136
  • Support K-EXAONE-2 in the PyTorch backend #17067
  • Add Qwen-Image-Layered with FP8 and Cache-DiT support #16889
  • Load static FP8 Cosmos3 Nano/Super checkpoints without re-quantization #17476
  • Enable EVS for Nemotron Super 3.5 VL and EVS handoff for Nemotron-H in EPD #19648, #19702
  • Load quantized Nemotron-H MTP heads #19558, #19597
  • Add MiniMax-M3 NVFP4 KV cache support with query-length buffer reuse and a native draft KV view #19422, #19768, #19731
  • Add MiniMax H3 ref2va mode and Turbo LoRA tensor-name mapping #19657, #19701
  • Route Qwen3.6/3.8 vision-language checkpoints to the generic Qwen3.5 VLM wrapper #19892

• API

  • Remove the Python backend of KVCacheManagerV2 (BREAKING) #19154
  • Migrate LlmRequest fields to Python (BREAKING) #19484
  • Add command-line configuration overrides #19555
  • Report per-request speculative-decoding acceptance in responses #19311
  • Return OpenAI-style errors when a request exceeds the context length #19457
  • Add a per-model serving-extension registry to the OpenAI server #19499
  • Ship the OpenEngine gRPC bindings in the wheel, so no separate package install is needed #19371

• Feature

  • Add a Mooncake store pool, CLI, and V2 scheduler preemption (part 1) #19235
  • Add sparse KV cache configuration, tier-aware locking, and offload with GPU metadata publication; prepare DeepSeek V4 attention for sparse KV offload #19580, #19821, #19502
  • Add VisualGen VSA and SOL sparse attention on PrimTS block-sparse FMHA and integrate updated SM90 SageAttention kernels #18079, #18496
  • Support beam search with the C++ KVCacheManagerV2 #17306
  • Stream C++ KV events with multimodal payloads #19359
  • Add model startup stage timing metrics, sampled iteration statistics, and partial telemetry context for startup failures #18060, #19717, #19598
  • Support model_kwargs with encoder CUDA graphs #19439
  • Speed up MoE routing for batches of up to eight tokens and cache MegaMoE kernel descriptors across forwards #19682, #19901
  • Enable tiled FP8 context FMHA for head dimension 256 on SM120 #19825
  • Add a static tile scheduler override for context FMHA #19873
  • Include KV connector prefixes in V2 scheduler budgeting and add configurable sliding-window attention endpoint retention priority #17974, #19670
  • Weight the speculative draft KV budget split by per-request spend #19744
  • Support post-SiLU clamp in CUTLASS MoE and add an optional finalize-before-combine path to the MoE scheduler #19217, #19749
  • Accept flat int32 arrays as GenerationRequest prompts without converting them to lists #19658
  • Allow the prefix-tokenization cache for MiniMax-M3 text-only prompts #19613
  • Reduce MTP draft-loop and multimodal fusion overhead #19785, #18636

• Fix

  • Fix host out-of-memory failures by releasing asynchronous IPC payloads when garbage collection is disabled #19869
  • Fix dummy-slot handling for Kimi K2.5 disaggregated EAGLE3 inference with attention data parallelism #19891
  • Harden disaggregated KV transfer: retire ownership after late completion, require verified write coverage before block reuse, and bound KV retirement #19378, #19545, #19673, #19674
  • Report cached_tokens from the prefix-reuse decision rather than the decode position #19743
  • Make seeded requests reproducible with speculative decoding #19601
  • Prevent multimodal scheduling stalls on the KV cache V2 scheduler #19801
  • Fix parsing of hyphenated Kimi K2 tool names, keep Markdown fences in DeepSeek V3 streamed content, and validate unused chat-template controls #19758, #19226, #19351
  • Serialize Hugging Face remote-code loading when building tokenizers #19553
  • Keep MPI progress alive on the pipeline-parallel sample-state relay thread #19643
  • Fix the torch.compile EAGLE3 acceptance length #19899
  • Guard the MiniMax-M3 FP8 indexer against padded cache slots and honor selector strides in MSA short-query attention #19113, #19709
  • Preserve OpenEngine conversation context and prefill handoffs, and release completed OpenEngine results when server garbage collection is disabled #19750, #19858
  • Tear down cleanly when multi-rank warmup fails and scope KV cache warmup capacity constraints to DeepSeek V4 #19456, #19213
  • Set NCCL environment variables before creating the NCCL unique ID #19780
  • Include the SM107 trtllm-gen context FMHA kernels #19626
  • Fix Kimi KDA prefill and decode synchronization and cap the Kimi K3 KDA prefill scratch caches #19581, #19830, #19829, #19885
  • Exclude padded Cosmos3 text from sequence-parallel attention and pin the Cosmos3 guardrail package #18705, #19504
  • Align LTX-2 scale factors with tensor axes #19500
  • Realign misaligned int32 index slices in FlashInfer GDN decode #17028
  • Preserve dense tensor layouts during offloading #18539
  • Skip unused multimodal encoder IPC handles #19791
  • Disable FA4 2CTA by default for VisualGen #19292
  • Prevent unintended dynamic NVRTC linkage through pg_utils #19156
  • Keep telemetry heartbeats running for long-lived sessions and report public runtime architectures #19620, #19704, #19595

• Documentation

  • Update documentation for NGC PyTorch 26.08 and PyTorch 2.14.0 #19019
  • Remove the outdated Kimi K3 limitation for supported SA speculation #19784

• Test & Infra

  • Ship a BOLT-optimized SBSA release wheel, and publish and test the post-merge BOLT build alongside the canonical build #19429, #19431, #19603, #19602
  • Remove Conan from the C++ build and improve the build structure #19552, #19445
  • Remove remaining TensorRT-backend leftovers and obsolete TensorRT telemetry compatibility paths #19583, #19585
  • Establish model runner contracts and migrate decoder execution into the runner family #19592, #19696, #19896
  • Update trtllm-gen FMHA artifacts #19591
  • Add VR200 4-GPU serving performance cases for DeepSeek V4 Pro and Kimi K3 #19260
  • Add KV transfer lifecycle and sliding-window attention eviction regression coverage #19663, #19754
  • Replace TinyLlama-1.1B-Chat-v1.0 with Qwen3-0.6B in tests #19178
  • Migrate disaggregated end-to-end tests to C++ bounce buffers #19588
  • Clean up orphaned performance test workers and isolate serving performance test logs #19697, #19930
  • Route documentation changes to docs and CPU validation #19409

What's Changed

• [None][fix] Fix quantized MTP head loading for Nemotron H by @mikeiovine in #19558
• [None][perf] Add MiniMax-M3 NVFP4 KV cache support by @peihu-nv in #19422
• [TRTLLM-16625][fix] Fix seeded request reproducibility on spec dec by @mikeiovine in #19601
• [TRTLLM-16304][feat] In-tree implementation of staircase by @WeiHaocheng in #19056
• [https://nvbugs/6801229][fix] Fix Rubin CuTe DSL MoE unit tests by @zhangcl in #19628
• [https://nvbugs/6786712][fix] Re-landed the previously-verified 3-defect payload tree-identically… by @trtllm-agent in #19427
• [None][chore] Remove dead code in model engine by @mikeiovine in #19513
• [TRTLLM-11412][fix] Preserve dense tensor layouts during offloading by @rahul-steiger-nv in #18539
• [None][feat] Add GLM-5.3-Flash (glm5_next) support by @ruocheng-nv in #19136
• [None][fix] Make integrator gain gate authoritative by @GuanhuaWang2001 in #19577
• [None][infra] Add scheduled CodeRabbit semantic checks by @chzblych in #19621
• [None][fix] Publish semantic reviews with per-PR commit statuses by @chzblych in #19649
• [None][doc] Update the docs for the NGC PyTorch 26.08 and torch 2.14.0 upgrades by @EmmaQiaoCh in #19019
• [#17027][fix] Realign misaligned int32 index slices in FlashInfer GDN decode by @Navjot10 in #17028
• [TRTLLM-12891][feat] Include KV connector prefixes in V2 scheduler budgeting by @eopXD in #17974
• [None][fix] Require integration evidence in semantic review prompts by @chzblych in #19651
• [None][feat] Add Qwen-Image-Layered FP8 and Cache-DiT support by @yumin066 in #16889
• [https://nvbugs/6835139][chore] Unwaive test_compiled_mxfp8_warmup_backend_selection by @mikeiovine in #19656
• [None][infra] Waive 1 failed cases for main in pre-merge 62247 by @trtllm-agent in #19662
• [https://nvbugs/6842831][fix] Restore FlashInfer GDN decode test collection by @pranav-nvidia in https://github.com/NVIDIA/TensorRT-LLM/pul...

Highlights

• Model Support

  • Support Qwen3.5 checkpoints with global FP8 scales across model loading paths #19519
  • Optimize Gemma4 vision rotary embeddings without a GEMM operation #19571
  • Enable the CuTeDSL MoE backend for Kimi K3 NVFP4 SiTU #19003
  • Expose Nemotron-H vision-language LoRA configuration for supported inference paths #19151
  • Extend MiniMax-M3 piecewise CUDA graphs through model and executor paths #19423

• API

  • Remove the AutoDeploy integration and its public entry points (BREAKING) #19028
  • Remove the obsolete cpp_only build option from CMake (BREAKING) #19421
  • Accept parallel_tool_calls and cap tool enum values in OpenAI requests #19459
  • Treat explicit null optional fields as unset in the Responses API #19454
  • Validate media data URIs and require opt-in for private URL fetches #19458

• Feature

  • Add an MLA-backed standalone DSpark drafter for Inferact and Kimi K3 #19040
  • Enable Helix speculative verification for FP8, FP4 MLA, and DSpark groups #19273
  • Add a non-MTP KDA decode kernel and dispatch path #18938
  • Add an NVFP4 MLA residual switch for DeepSeek V4 #19305
  • Support MNNVL all-reduce through the Ray orchestrator #18231
  • Add opt-in cached KV token counts and per-rank routing logs #19375, #16323
  • Add quiet PyExecutor hang diagnostics for stalled inference requests #19045
  • Support backend-independent agent tools and a required-tool policy #19236
  • Update scaffolding MCP transport and KV-cache truncation behavior #19055
  • Support W4A16 NVFP4 group size 32 with explicit validation #19091
  • Add Triton top-k combine for Marlin NVFP4 MoE #18999
  • Fuse MLA context FP8 Q quantization into the absorb BMM epilogue #19082
  • Enable PDL on additional decode edges and FlashInfer GDN prefill on SM120/121 #19446, #19376
  • Defer fused-GELU MLP NVFP4 scale finalization until needed #18906
  • Avoid KV-cache V2 prefix probes when context chunk budget is insufficient #19366
  • Publish MoE all-to-all counters before dispatch and consolidate C++ MoE ops #19425, #19216

• Fix

  • Compute Cosmos3 rotary tables with broadcast multiplication to avoid K=1 matmul failures #19532
  • Restore Qwen3.5 gather and scatter behavior after a performance regression #19578
  • Preserve disaggregated KV transfer outcomes and synchronize receive destinations #19377, #19444
  • Authenticate disaggregated subagent affinity and reject unbindable IPv6 listener addresses #19175, #19544
  • Make disaggregated load-balancing routing global and repair cache-reuse adapter behavior #19399, #19480
  • Extend startup and warmup timeouts for Blackwell pipeline-parallel tests #19402
  • Keep one-sided MoE combine offsets and FP4 permute buffers correctly sized #19312, #19251
  • Reject paged-context attention when no fused kernel exists #19353
  • Share speculative CUDA-graph capture buffers and correct RNG corner cases #19350, #19505
  • Correct DFlash buffer sizing, positions, and FlashInfer speculation gating #19466
  • Guard PyExecutor model-engine access while handling responses #19559
  • Route zero-token causal convolution through the channel-major kernel #18571
  • Correct VBWS width tracking and terminal-step finalization #19407
  • Drain guided-decoder host functions before forward calls that hold the GIL #19455
  • Prevent out-of-bounds indices in vanilla top-p sampling #19496
  • Reuse pending MNNVL split communicators across workspace construction retries #19101

• Documentation

  • Document GVR V2 self-sampling and multi-threshold exact top-k methods #19244

• Test & Infra

  • Upgrade public PyTorch to 2.14, Triton to 3.8, and C++ to 20 #19477, #19249
  • Add Helix zero-KV, disaggregated closure, and speculative acceptance regression coverage #18995, #19478, #19176
  • Expand Qwen3.8 coverage across GPU configurations #19530
  • Disable context overlap for the DeepSeek V4 Pro disaggregated test configuration #19424
  • Fix GPT-OSS Eagle3 disaggregated fixtures and DeepSeek V4 assertions #19531, #19554
  • Correct DFlash budget, MPI FlashInfer, and MNNVL memory test fixtures #19582, #19254, #19509
  • Set bounded retries for VisualGen ffmpeg installation in tests #19433
  • Correct Wan 2.2 LPIPS thresholds and restore its accuracy test #19438
  • Preserve CI logs while detecting GCP hosts and identify SLURM job origins #19345, #19327
  • Key the legacy lint baseline on POSIX paths #17745
  • Update telemetry code ownership for configuration review #19599
  • Regenerate the LLM-args telemetry manifest for sparse-attention configuration #19493
  • Remove deprecated GPT-OSS Hopper tests and obsolete TensorRT leftovers #19470, #19428

What's Changed

• [TRTLLM-16564][feat] MLA-backboned standalone DSpark drafter (Inferact/Kimi-K3-DSpark) by @dc3671 in #19040
• [None][fix] Responses API: treat explicit null as unset for optional fields by @brnguyen2 in #19454
• [None][perf] defer dynamic NVFP4 scale finalization for fused GELU MLPs by @daichu-nv in #18906
• [None][fix] Regenerate LLM-args golden manifest for sparse_attention_config.uses_spcompress by @brnguyen2 in #19493
• [None][feat] agent-flow: backend-independent tools and required-tool policy by @kaiyux in #19236
• [None][feat] Kimi K3: unlock the CUTEDSL MoE backend for NVFP4 SiTU by @xguannv in #19003
• [None][fix] Share speculative capture buffers across CUDA graph buckets by @brnguyen2 in #19350
• [None][fix] Validate media data-URIs and gate private-URL fetches behind an opt-in by @brnguyen2 in #19458
• [None][perf] Avoid prefix probes when the KV cache v2 context chunk budget is insufficient by @xwang233 in #19366
• [https://nvbugs/6708299][fix] Support group_size=32 for W4A16_NVFP4 and validate NVFP4 group sizes by @moraxu in #19091
• [https://nvbugs/6649739][fix] Skip deprecated GPT-OSS Hopper tests and remove stale waivers by @dongfengy in #19470
• [None][feat] Helix speculative verify groups: fp8 + fp4 MLA and DSpark by @reasonsolo in #19273
• [None][test] Disable ctx overlap scheduler for deepseek-v4-pro con8 disagg config by @chenfeiz0326 in #19424
• [None][feat] NVFP4 MLA residual switch for DeepSeek-V4 by @Tracin in #19305
• [TRTLLM-16398] [test] Add disaggregated speculative decoding acceptance-length baselines by @allisonlim-nv in #19176
• [TRTLLM-15820][feat] Expose Nemotron-H VL LoRA configuration hook by @eopXD in #19151
• [TRTLLM-15899][feat] Add new KDA decode non-mtp kernel and dispatch logic by @pengbowang-nv in #18938
• [https://nvbugs/6607482][logging] Add quiet PyExecutor hang diagnostics by @2ez4bz in #19045
• [https://nvbugs/6705034][fix] Route zero-token causal-conv input to the channel-major kernel by @nv-guomingz in #18571
• [TRTLLMINF-440][infra] Upgrade C++ from 17 to 20 by @EmmaQiaoCh in #19249
• [TRTLLM-16394][refactor] Consolidate and group the cpp MoE torch ops under thop/moe/ by @lori-ren in #19216
• [https://nvbugs/6758990][fix] Fix MPI flashinfer test failure by @tongyuantongyu in #19254
• [None][doc] GVR V2: Self-Sampling and Multi-Thresholding for Faster Exact Top-K by @longcheng-nv in #19244
• [#19337][fix] Correct VBWS width tracking and terminal-step finalization by @yifanQi98 in #19407
• [None][fix] make disagg load balancing router global by @reasonsolo in #19399
• [None][fix] fix _CacheReuseAdapterV1 by @bo-nv in #19480
• [TRTLLMINF-420][fix] Add Jenkins instance name to SLURM job by @lyxxn0414 in #19327
• [https://nvbugs/6777522][test] Guard VisualGen ffmpeg apt install with timeout and retry by @zhenhuaw-me in #19433
• [https://nvbugs/6813629][fix] Fix GPT-OSS Eagle3 disaggregated test configuration by @zhaoyangwang-nvidia in #19531
• [None][perf] PDL on the missing decode edges by @brb-nv in #19446
• [None][fix] size FP4 block-scale MoE permute buffers from local_num_experts by @reasonsolo in #19251
• [#19487][fix] Fix CUDA graph spec dec RNG corner case by @mikeiovine in #19505
• [TRTLLM-15718][test] Add disagg closure tests, demote one E2E to post-merge and replace another with CPU/1-GPU tests by @nv-xtf in #19478
• [None][perf] MLA context: fuse the Q FP8 quantization into the absorb bmm epilogue (CuTe-DSL BF16->FP8 bmm) + RoPE kOutputFp8Q by @Tabrizian in #19082
• [None][feat] OpenAI protocol: cap tool enum values and accept parallel_tool_calls by @brnguyen2 in #19459
• [None][fix] Give test_mnnvl_memory an expert-parallel mapping and assert its checks by @brnguyen2 in http...

Highlights

• Known Issues

  • Multi-node inference may fail during startup when a requested parallel mapping spans nodes or request payloads reference caller-local Python modules.
  • Disaggregated serving may fail to start when an advertised hostname does not resolve to a bindable IPv4 address. Use a hostname that resolves to a local IPv4 interface.
  • Qwen3.5-4B may lose accuracy with context pipeline parallelism and a different generation tensor-parallel layout.
  • Overlapped TinyLlama disaggregated serving may return content associated with another prompt in the same request batch.
  • Eight-rank asymmetric NIXL KV-cache transfers may fail intermittently, preventing disaggregated cache exchange.
  • MiniMax-M3 MXFP8 inference with MSA and piecewise CUDA graphs may crash during executor initialization.
  • Nemotron-3 Super FP8 inference may lose accuracy when using the C++ Mamba cache without attention data parallelism. Use the Python Mamba cache to avoid the affected path.
  • DeepSeek-V4-Flash NVFP4 inference may intermittently crash with an illegal memory access while initializing the KV cache.
  • Qwen3-30B-A3B skip-softmax attention may hang with 0.9 sparsity and FP8 KV cache.
  • Qwen3-VL forced chunked prefill may intermittently terminate multimodal requests before generation.
  • Two-rank tensor-parallel execution may intermittently abort while importing CUDA IPC memory handles.

• Model Support

  • Extend Qwen3.8-Flash-Next with disaggregation, MegaMoE, and GVR index reuse #18921, #19323, #19138
  • Enable MiniMax-M3 Eagle3 speculation and MegaMoE CuTeDSL execution #18872, #18605
  • Optimize Kimi-K3 attention-residual RMSNorm fusion and KDA cache alignment #19182
  • Enable Nemotron multimodal KVCM2 and Transformers 5.13 checkpoint loading #19140, #19168
  • Expand Qwen-Image attention backends and Cosmos3 Edge Diffusers parity #18147, #19097

• API

  • Remove deprecated TensorRT serve, evaluation, benchmark, and stress paths (BREAKING) #19022, #19021, #18850
  • Remove the deprecated Eagle choices configuration field (BREAKING) #18210
  • Remove unused LlmRequest methods and legacy request code (BREAKING) #18977, #18902
  • Deprecate the Triton MoE backend in favor of supported implementations #19035
  • Remove callable telemetry schema metadata support from configuration capture #19360

• Feature

  • Add DFlash 2 support with model-specific MoE backend overrides #18155, #17408
  • Enable KV-cache manager V2 by default for Llama and Llama4 #19004
  • Track KV-cache reuse by source tier and extend connector support #18583, #18762
  • Unify native KV transfer coordination, admission, and backend contracts #19128, #19266, #19198
  • Add token-aware ADP routing and enhanced NIXL transceiver buffering #19130, #15780, #18939
  • Integrate Sol-Attn sparse attention into VisualGen pipelines #18329
  • Give VisualGen servers isolated media directories for concurrent requests #19135
  • Add FP8 KV-cache support to PrimTS MLA decode #18946
  • Add NCCL-EP 0.2 low-latency expert-parallel integration #18689
  • Return per-token MoE routing through Router Replay to training #18397
  • Add fused sampling and min-p to AdvancedSamplingMode.FULL #18515
  • Validate CUDA-graph input-buffer readiness before replay #18207
  • Optimize MiniMax-M3 QKV, MSA writes, and CuTeDSL autotuning #18205, #18614, #18620, #17015
  • Reduce DeepEP metadata overhead and tune MegaMoE for Blackwell #16919, #17956
  • Tune GVR, DSA, and PrimTS kernels across Blackwell generations #19076, #19094, #19096
  • Cache input-prefix tokenization and reduce default NumExpr thread overhead #18389, #18556
  • Deduplicate VisualGen Wan RoPE shared-memory staging #18036
  • Extract encoder runners and reorganize disaggregated beam-search support #18933, #18638
  • Add FlashInfer decode overrides with complete autotuner candidate logging #19162
  • Centralize multimodal rotary dummy-slot state across execution paths #19290
  • Optimize NVFP4 MLA gathering and profile dense cached KV chunks #19291, #19269
  • Preallocate GDN prefill state workspaces for repeated execution #19108
  • Add attention-kernel failure context and optional page-table diagnostics #18756
  • Consolidate MoE components and simplify Kimi FP8 checkpoint loading #19061, #19179

• Fix

  • Correct KVCM2 quotas, block scales, parity, admission, and draft sizing #18988, #19192, #19103, #18697, #19018, #19202
  • Share speculative draft-length updates during warmup and capacity planning #19204
  • Correct V2 KV-cache estimation for GPT-OSS and related workloads #18910
  • Stabilize XQA attention-sink normalization for fully masked partial rows #19220
  • Fix mixed-precision MoE configuration, LoRA propagation, and Ray workspaces #18393, #19109, #18300
  • Reject unsupported one-model speculative parameters before streaming begins #19161
  • Restore grammar state across speculative verification modes #19296
  • Default unset non-greedy speculative temperatures and avoid unused draft caches #19390, #19410
  • Convert DFlash capture buffers to the correct runtime dtype #19354
  • Stop charging retiring requests against ADP admission capacity #18457
  • Preserve integrated-GPU host-tier budgets during cache sizing #18926
  • Load custom tokenizers consistently and reject empty completion prompts #18680, #17737
  • Preserve Cosmos3 metadata and make MoonViT replication helix-aware #18909, #19277
  • Correct Wan fused layer normalization with FP32 accumulation #17987
  • Fix NVLink discovery below probe bounds and release attention workspaces #19210, #19051
  • Correct RocketKV cache sizing and avoid full Triton prefill casts #19264, #19385
  • Add SM100-family JIT support for MiniMax sparse attention #19372
  • Preserve logger formatting arguments and external wheel requirements #19160, #19286
  • Reconfigure CMake when wheel-build arguments change #19339
  • Preserve multimodal KV-cache keys across V2 event publication #18810
  • Compose telemetry capture policies across nested configuration fields #18978
  • Accept FlashInfer ultra-wide W4A16 tiles and isolate rank-local cubins #18898, #19034
  • Honor every greedy stop token through canonical termination checks #19049
  • Preserve mixed-BF16 FlashInfer attention dispatch #19144
  • Keep strict Qwen3 LoRA validation with KV-cache manager V2 #19206
  • Match vendored include patterns by complete path components #19274
  • Warn when DFlash is combined with unsupported disaggregated serving #19352

• Documentation

  • Document DeepSeek-V4 support in NVFP4 cold-page compression #19317
  • Remove the superseded deployment guide from published documentation #19426
  • Keep historical change rationale in pull-request descriptions #19419
  • Restore bounce-buffer capture metadata during documentation builds #19279
  • Clarify phased FMHA inputs, outputs, and ownership contracts #19284
  • Document CUDA-graph padding bounds and tail-only invariants #19163

• Test & Infra

  • Upgrade public PyTorch and Triton while supporting DLFW release wheels #18743, #18628
  • Clean development images and derive DLFW wheel versions automatically #19152, #19248
  • Add CBTS shadow runs and Python change-reference analysis #18814, #19124
  • Expand model coverage for Phi-3, Exaone4, Whisper, and DeepSeek-V3.2 #19245, #19246, #19256, #19332
  • Make missing VisualGen checkpoints and dependencies fail explicitly #19054
  • Publish canonical BOLT profiles and disable unsupported aggregate cases #18615, #19299
  • Improve confidentiality scanning, internal Git fallback, and stalled-job cleanup #19200, #18951, #19165
  • Add InferenceX-style GSM8K evaluation and model-feature qualification #18738, #19046
  • Refine pre-merge performance gating and generation-only sanity runs #18766, #19107
  • Add power-law MoE routing controls and additional DFlash coverage #19148, #19159
  • Register LTX-2 VAE and executor IPC validation in CI #19203
  • Enforce V2 KV-cache iteration statistics and parity prewarming #19207, #19132
  • Stabilize AgentX timeouts, MoE-LoRA measurement, and MiniMax fixtures #19219, #19255, #19298
  • Gate SM103 graph tests and cap hybrid-model batch sizes #19280, #19398
  • Bound MiniMax serving configurations and trim oversized perf-sanity cases #19344, #19349
  • Initialize KV-cache fixtures and restore deterministic test coverage #19382
  • Remove obsolete K2, DeepSeek, Llama, and flaky parity coverage #19084, #19114, #18654, #19239
  • Stabilize attention rotary, MSA layout, and backend-policy tests #18774, #19078, #18811
  • Update performance helpers and balance checkpoint-I/O experiment arms #19129, #19170
  • Remove deprecated MiniMax coverage and correct stop-word expectations #19030, #19158
  • Cover MiniMax C++ NIXL bounce and Nemotron graph parity #19288, #19335

What's Changed

• [None][infra] Skip pre-merge perf gating when main has already regressed and refactor the pre-merge perf-sanity list by @chenfeiz0326 in #18766
• [https://nvbugs/6758853][fix] Prewarm prefixes for KV cache parity comparisons by @yizhang-nv in #19132
• [TRTLLM-15788][feat] Add dflash 2 support by @mikeiovine in #18155
• [TRTLLM-14093][feat] Eagle3 support for MiniMax-M3 by @zheyuf in #18872
• [None][chore] BREAKING: Remove TRT leftovers from serve and eval by @Funatiq in #19022
• [https://nvbugs/6556429][fix] Fix host tier budget on integrated GPUs by @pamelap-nvidia in #18926
• [None][chore] Log attention kernel failure context and add an opt-in host-side page-table check by @brnguyen2 in #18756
• [https://nvbugs/6661846][fix] Initialize...

Highlights

• Known Issues

  • GPT-OSS Eagle3 speculative decoding may hang, produce non-terminating output, or lose accuracy when using multi-GPU execution.
  • GPT-OSS W4 inference with the Triton MoE backend may intermittently hang or lose accuracy in four-GPU data-parallel or expert-parallel configurations.
  • GPT-OSS Harmony tool calls may continue generating after the tool-call delimiter, resulting in malformed or repeated tool calls and an incorrect finish reason.
  • Qwen3-235B NVFP4 inference on B200 may intermittently lose accuracy when all-reduce autotuning is enabled. Set TLLM_DISABLE_ALLREDUCE_AUTOTUNE=1 to avoid the issue.
  • Qwen3-6B-A3B NVFP4 W4A16 inference may fail during initialization on Blackwell SM120 and SM121 GPUs.
  • MiniMax-M3 MXFP8 inference using piecewise CUDA graphs with the Triton sparse-attention implementation may crash during initialization on B300 and GB300 systems. Use the MSA implementation to avoid the affected path.
  • Kimi-K3-NVFP4 may fail to initialize its tokenizer with the Transformers version included in DLFW 26.08.
  • Nemotron-H weight updates through Ray may OOM or produce incorrect logits with DLFW 26.08 because compatible Mamba dependencies are unavailable.
  • mBART encoder-decoder inference may hang during the first generation request with DLFW 26.08.
  • Cosmos3 pipelines may fail to load when VAE CPU offload is enabled. Disable VAE CPU offload to avoid the affected path.
  • Multi-GPU disaggregated serving may hang during worker startup for TinyLlama deployments using pipeline or tensor parallelism, and for GPT-OSS-120B generation workers using TP4 on GB200.

• Model Support

  • Add MiniMax-H3 VisualGen support for text-to-video with audio and first/last-frame-to-video with audio, with BF16 and FP8 blockwise configurations #18733
  • Enable Nemotron 3.5 Super VL serving with multimodal model delegation and automatic reasoning and tool-parser selection #18804
  • Add Cosmos3 Edge Policy DROID support for state-conditioned robot-policy rollout and action generation through VisualGen and trtllm-serve #18463

• API

  • Remove deprecated two-model speculative decoding and related configuration fields (BREAKING) #18721
  • Remove unused legacy C++ runtime, batch-manager APIs, and Python bindings (BREAKING) #18846
  • Add Anthropic-compatible Messages, token-counting, and Message Batches APIs to trtllm-serve #18289
  • Add LLM profiling methods and serving endpoints across supported execution modes #13872
  • Enable streaming OpenEngine Generate RPC with disaggregated-serving request and control integration #17992

• Feature

  • Integrate PrimTS FMHA and block-sparse attention with unified runtime inputs #17399, #18815
  • Add cuDNN as a selectable attention backend for supported workloads #18075
  • Add FlashInfer VisualGen attention and FP4 Conv3d for parallel Wan VAE #18174, #17262
  • Support Ray-orchestrated postprocessing workers for efficient distributed output handling #18399
  • Preserve sub-agent conversation affinity across disaggregated-serving requests and workers #18684
  • Add optional KV guard pages and fresh-page-fill diagnostics for debugging #18754
  • Derive per-layer KV-cache windows from hybrid model layer types automatically #18767
  • Publish KV-cache events through StreamingEvent for higher-throughput subscriber delivery #17023
  • Admit one context per uncached prefix, preventing duplicate prefix recomputation #18195
  • Add in-graph temperature, top-k, and top-p sampling for batched requests #18437
  • Add Kimi K3 KDA replay using KV-cache manager V2 #18294
  • Page DSpark drafter contexts through the dedicated draft KV-cache manager #18343
  • Add GVR V2 self-sampling prefill and two-level decode top-k dispatch #18702, #18446
  • Support image inputs through the Triton LLM API backend #18381
  • Optimize Qwen3.5 and Qwen3.8 Flash-Next generation, MoE, GDN, and sampling #18553, #17365, #18883, #18714, #18820, #18845
  • Accelerate long-sequence KV-cache resizing and asynchronous PP1 disaggregated transfers #18541, #18150
  • Optimize MiniMax-M3 sparse decode, HCA decode, and NVFP4 Marlin normalization #18611, #18869, #17685
  • Apply NUMA-aware CPU affinity across serving worker threads and processes #18701

• Fix

  • Prevent generation workers from re-expanding multimodal placeholders during disaggregated serving #18886
  • Warn when tool-call markup cannot be parsed into a valid call #18865
  • Enable breakable prefill CUDA graphs for Nemotron-H hybrid models #18888
  • Fix KVCM V2 buffers, vocabulary sizing, page tables, draft reuse, and uploads #18824, #18957, #18885, #17643, #18093, #18694
  • Re-plan FlashInfer after page-table changes and bound eager plan-cache growth #18769, #18557
  • Stabilize MoE LoRA CUDA-graph storage and propagate per-layer adapter metadata #18972, #18527
  • Correct SM107 FMHA, FP8 GEMM, and NVFP4 KV-cache capability routing #18758, #18945, #18612, #18674
  • Fix quantized MoE sharding, providers, scales, and Kimi K3 SiTu formats #18870, #18770, #18772, #18709
  • Restore DSA TopK workspace lifetimes, row sizing, and warmup behavior #18624, #18683
  • Fix XQA synchronization and attention-sink numerical accuracy during decode #18141
  • Harden distributed execution across UserBuffers, VMM-backed DWDP, and MNNVL layouts #18037, #18644, #18715
  • Correct speculative token accounting and Eagle3 KV block-table rotation #18629, #18564
  • Fix disaggregated KV ownership, relay sends, MTP accuracy, and launch ordering #18041, #18135, #18554, #18613
  • Align generation-only request detection across Python, C++, scheduling, and disaggregation #18858
  • Correct custom-tokenizer alias resolution during model initialization in LLM arguments #18768
  • Densify unsupported SM120 BMM inputs and restore FlashInfer MLA dispatch #18678, #18653
  • Release MegaMoE buffers during teardown and page out DeepSeek-V4 weights #18182, #18832
  • Share Kimi auxiliary streams safely across model components during execution #18728
  • Fail guided-decoding requests cleanly when grammars reach dead-end states #18896
  • Stabilize VisualGen randomness, quantization, worker cleanup, and serving clocks #18477, #18750, #18030, #18686
  • Load mixed-precision Qwen3.8-Flash-Next NVFP4 checkpoints correctly across supported backends #18823
  • Support MoE two-sided preparation beyond 128 expert-parallel ranks in large deployments #18799
  • Preserve configured Mamba snapshot placement across runtime initialization and cache reuse #18724
  • Install built wheels using the active virtual-environment interpreter during development workflows #18773
  • Match fused finish-reason checks to the sampler's actual scratch row #18894
  • Update stale attention and KV-cache imports blocking backend initialization #18771
  • Improve errors for invalid roles, missing wheels, and EBADHANDLE config reads #17730, #18703, #18352
  • Update vendored PrimTS attention with FlashInfer follow-up corrections for stability #18996

• Documentation

  • Add KV-cache compression guides, configuration guidance, and runnable user examples #18587
  • Add VisualGen quantized-attention, sparse-attention, and CUDA-graph implementation guides for developers #18792, #19023
  • Expand sparse-attention documentation with MQA and GQA developer guidance #18106
  • Document generation-side timing segments used in serving performance benchmarks #18213
  • Correct serving commands, configuration instructions, and broken example snippets #18500, #18901, #19009
  • Repair documentation links and images while reducing Sphinx build warnings #18984, #18992, #19007, #19050

• Test & Infra

  • Upgrade build and testing infrastructure to NGC PyTorch 26.08 dependencies #18126
  • Expand coverage for BART, DeciLM, guided/speculative decoding, Kimi, Qwen, and Gemma #19017, #18948, #18908, #18905, #18699, #18655
  • Add performance coverage for Qwen, DeepSeek, Kimi, and disaggregated deployments #19012, #18853, #18875, #18806, #18633
  • Refine VisualGen serving benchmarks and performance-sanity validation coverage across models #18632
  • Add disaggregated lifecycle telemetry and checkpoint-startup measurements to performance tests #18990, #18608, #18856
  • Extend performance automation with parallel optimization, remote execution, and timeline analysis #18568, #18789, #18805, #18825
  • Align Hopper and multi-node disaggregated coverage with Python transceiver runtime #18941, #18828, #18404
  • Improve CI attribution using typed cleanup failures, timeouts, reporting, and retries #18049, #18812, #18590, #17630, #17069
  • Stabilize batching, Eagle VSWA, Kimi streams, LoRA scheduling, and LPIPS tests #19005, #18986, #19088, #18949, #18386, #18384
  • Add strict documentation-link and declarative-import validation coverage across CI stages #18718, #18586
  • Modernize CI with GB300 flex scheduling, deferral, Artifactory images, and labeling #19074, #18385, #18791, #18729

What's Changed

• [TRTLLM-15011][infra] Unwaive TestDeepSeekV4Flash::test_auto_dtype by @lowsfer in #18000
• [None][refactor] Table-drive the lazy-safetensors model-type check in HfWeightLoader by @moraxu in #18681
• [https://nvbugs/6593378][baseline] Overwrite partial Slurm artifact retries by @BowenFu in #17630
• [https://nvbugs/6674826][fix] Drop overlap hint from KV cache page-index upload by @lowsfer in #18694
• [NVBUG-6689820][test] unwaive Llama disaggregated serving test by @chienchunhung in #18695
• [https://nvbugs/6683840][fix] Join the encoder launch before the decoder forward by @pranav-nvidia in #18613
• [None][feat] Add SM107 NVFP4 C...

Highlights

• Known Issues

  • LoRA adapters that target the routed experts of Qwen3 mixture-of-experts models are silently ignored and produce base-model output. Adapters targeting attention layers behave correctly.
  • Models using the V2 KV cache manager may fail to start with an insufficient-memory error even when enough GPU memory is free, because the KV cache size estimate is too conservative. GPT-OSS with Eagle-3 speculative decoding and sliding-window cache reuse is the most affected configuration. Lowering the maximum batch size or raising the free GPU memory fraction can allow startup to proceed.
  • Long-running servers using the FlashInfer attention backend may run out of GPU memory over time as attention planning resources accumulate across varying batch sizes. Using the default TRTLLM attention backend avoids this.
  • Image generation and image editing requests to a VisualGen deployment can fail with an HTTP 500 error, and video endpoints report inconsistent timing values. Text generation is unaffected.
  • Disaggregated serving requests may fail with a server error after a transient network interruption between the router and a context or generation worker, because the retried request is no longer matched to its in-flight counterpart.
  • Models using multi-head latent attention, such as DeepSeek, may hang or crash during generation on SM120 Blackwell GPUs, most often with multi-token prediction and chunked prefill enabled.
  • NVFP4 W4A16 mixture-of-experts models are unsupported on SM120 Blackwell GPUs and abort while preparing kernels. A future release will address this issue.
  • Enabling online expert-parallel load balancing with the TRTLLM-gen mixture-of-experts backend on unquantized models fails during model loading. The CUTLASS mixture-of-experts backend supports the same configuration.
  • Diffusion pipelines that offload the VAE component, such as Wan and Cosmos3, can fail to load. Running without VAE offloading works.
  • Kimi KDA models may crash when a scheduled batch contains no prefill work, which can happen under mixed prefill and decode scheduling.
  • Creating two models with different mixture-of-experts expert-parallel layouts in a single process fails. Use a separate process per configuration.

• Model Support

  • Add DeepSeek-V4 Hopper support #16940
  • Add Qwen3.8-Flash-Next support #18585
  • Enable Kimi K3 on B200 #18164
  • Enable Nemotron Super 3.5 VL NVFP4 and video input #18526, #18370
  • Qualify Mistral and Qwen3 dense models for Model Express #17142, #18558

• API

  • Remove TRTLLMSampler (BREAKING) #18232, #18233, #18532
  • Redesign VisualGen reference inputs and worker IPC (BREAKING) #17493
  • Complete the /v1/responses surface #18130
  • Add Kimi K3 API compliance in trtllm-serve #17845
  • Add authentication for RL control endpoints #17434
  • Support response_format='path' on /v1/images/edits #18444

• Feature

  • Add Rubin SM107 GEMM, MoE, and CuTe DSL support #17485, #17707, #18197, #18369
  • Support custom masks in TRTLLM attention #16214
  • Add SkipSoftmax sparse attention to the CuTe DSL backend #17781
  • Enable CuTe DSL MLA with Helix #18131
  • Make MLA skip correction configurable #18118
  • Add an MLA post-attention o_proj gate hook #18375
  • Update trtllm-gen FMHA kernels and cubins #18183
  • Cache and extract FMHA selection in TrtllmAttention #18350, #18548
  • Add encoder CUDA graphs for fixed-shape encoders #17030
  • Refactor advanced-sampling CUDA graph capture #17936
  • Support breakable CUDA graph for the Kimi K3 KDA mixer #17816
  • Add NVFP4 KV cache for DSA and cold-page compression #17681, #18091
  • Add Mamba state snapshots for hybrid KV cache reuse #18272
  • Improve KV cache manager V2 stats and VMM allocation #16710, #18347
  • Make the Python KV cache transceiver the default #18134
  • Add pipelined KV cache transfer for disaggregated serving #15727
  • Add a DisaggTransferCoordinator and transceiver conformance fake #18595, #18178
  • Split connector KV save out of _send_kv_async #18186
  • Use msgspec msgpack for disaggregated worker bodies #16873
  • Add a selectable OpenEngine gRPC server #17084
  • Add exit code telemetry to the CLI and LLM API #17377
  • Consolidate DSpark draft paths and support standalone drafters #18043
  • Add emission-assisted GVR top-K decode for DeepSeek V4 #16953
  • Improve GVR V2 decode top-k and prior seeding #18410, #18646
  • Optimize DSA decode metadata, index remap, and paged MQA #18268, #18391, #18383
  • Add an env-gated BF16 DSA indexer projection #18264
  • Enable fused-A GEMM for the GLM MLA A-projection #18265
  • Optimize DSpark attention with Blackwell CuTe DSL kernels #18061
  • Optimize Kimi K3 weight loading and prefill metadata #17741, #18064, #18643
  • Optimize Kimi KDA decode and prefill convolution #18251, #17870, #17862
  • Convolve the Mamba2 prefill on a channel-last projection #18081
  • Optimize MiniMax-M3 GEMMs, decode q, and block selector #17238, #18474, #18154
  • Optimize Qwen3.6-35B-A3B-NVFP4 on SM120 #17976
  • Add split-K auto-tuning to the LoRA grouped GEMM #17315
  • Support uneven split-K and fix fused mHC coherence #18331
  • Fuse relu2 into a single kernel #18082
  • Patch DeepEP for top-k 16 #17984
  • Make the DeepGEMM barrier timeout configurable #18179
  • Add a locality domain Python layer #18317
  • Preserve MNNVL and MoE A2A graph addresses across restore #16635, #16632
  • Reuse FlashInfer plans for single-token decode #18596
  • Merge attention-DP host collectives #18293, #18639
  • Implement rank-striped checkpoint read-ahead #17393
  • Forward speculative-decoding state through the Nemotron VL wrapper #18647
  • Add offloading support for VisualGen #14095
  • Add VisualGen FP8 row-wise quantization #16847
  • Add Cosmos3 action generation to VisualGen #17325
  • Add a trtllm-eval pipeline for VisualGen #18199
  • Declare MoE backend capabilities instead of class checks #18409
  • Relocate MoE code under _torch/moe/ #17952
  • Extract multimodal encoder scheduling into engine/multimodal.py #18524
  • Remove unused KV cache draft-token rewind helpers #18533

• Fix

  • Isolate FlashInfer JIT workspaces across MPI processes #18278
  • Normalize perf metrics clocks across frontend processes #17491
  • Guard CuTe DSL grouped-GEMM imports in fused MoE #18144, #18142
  • Bump CUTLASS DSL to 4.6.2 for FA4 split-KV #18145
  • Include the FINALIZE-fusion workspace in the MoE autotuner #18675
  • Size the SM90 fp8 block-scale MoE workspace correctly #18323, #18322
  • Reject NVFP4 MoE layers with unaligned gated FC1 #18520
  • Reject incompatible cached low-M GEMM tactics #18635
  • Support MXFP8 under MiniMax SwiGLU in CutlassFusedMoE #18257
  • Pass the runtime SM to the NVFP4 Marlin support check #17502
  • Fix barrier-divergence races in the GVR top-k decode kernel #18339
  • Fix self-sampling top-k in the register family #18625, #18501
  • Reject unsupported speculative outputs #18401
  • Use prompt lookahead for MTP Eagle chunked prefill #18295
  • Keep CuTe DSL MLA decode for Kimi K3 H=96 verify batches #18226
  • Apply the K3 prompt-token offset only when known #18509
  • Keep quantized KV context on TRTLLM-Gen FMHA #18545
  • Use fallback FMHA for SM103 context attention #18354
  • Fix Gemma4 FMHA fallback and video token counting #18544, #18274
  • Fix window vector layer indexing #17466
  • Fix MiniMax-M3 MSA KV cache mapping and vision meta init #17374, #18466
  • Remove an invalid GLM KV cache assertion #18374
  • Support fixed-only draft KV cache budgets #18459
  • Exclude mamba conv1d from quantization #18607
  • Fix Qwen3.8-27B NVFP4 checkpoint loading #18387
  • Fix LLM-only behavior for Mistral Large #18086
  • Fix attention-DP dummy token accounting #18308, #18157
  • Register beneficial-to-skip contributions after scheduling #18202
  • Recalculate PEFT cache capacity for the adapter dtype #18200
  • Guard Python KV receive ownership and publication #17720
  • Avoid sleeping out the KV transfer poll interval #18175
  • Keep disaggregated worker liveness within its TTL #18403
  • Fail-stop on distributed warmup errors #17667
  • Stabilize the telemetry lifecycle #18454
  • Fix lazy imports for profile matching and MPI worker submit #18236, #18284
  • Restore verl rollout after the sampler_type removal #18513
  • Restore Wan 5B per-token AdaLN with TeaCache #18209
  • Preserve prepared Cosmos3 video frames #18464
  • Allow sm_107a in the VisualGen CuTe DSL FMHA #18476
  • Raise the Starlette security floor #17650

• Documentation

  • Split the LLM API reference into smaller pages #18440
  • Document the FMHA availability contract #18710
  • Update the GLM-5 docs for GLM-5.3 and disaggregated serving #18388
  • Fix stale Wide-EP EPLB documentation #18626
  • Clarify nightly release installation #18282
  • Fix the multimodal quick start clone step #18429
  • Give a runnable lm_eval install command #18431
  • Fix XGrammar structural tag doc links #18430
  • Remove the invalid html_inline myst extension #18427
  • Add a human-in-the-loop section to the modeling-bringup README #18358
  • Add a video generation optimization blog #16506

• Test & Infra

  • Update flashinfer-python to 0.6.18 #18428
  • Declare and pin CI-imported dependencies #18033
  • Add a declarative extra import path for test sources #17884
  • Upload disaggregated lifecycle spans to OpenSearch #18445
  • Key perf-sanity case identity on the test case name #18408
  • Adjust perf-sanity ctx_only scheduling and warmup #18346, #18432
  • Add AgentX DeepSeek-V4-Pro-DSpark perf lanes on GB300 #18298
  • Add MiniMax-M3 disaggregated perf recipes #18416
  • Add GB300 multi-node Qwen3.8-2.4T-A95B accuracy stages #18363
  • Add Kimi K3 accuracy and speculative decoding tests #17922, #17921
  • Add KimiLinearForCausalLM helix ma...

Highlights

• Annoucements

  • KV Cache Manager
    • Enabled KV cache manager V2 by default for:
      • DeepSeek V3, DeepSeek R1, and DeepSeek V3 Lite
      • DeepSeek V3.2
      • DeepSeek V4
      • GLM-5
      • GPT-OSS
      • Mistral Large 3
      • Kimi K2, Kimi K2.5, and Kimi K3
      • MiniMax M2 and MiniMax M3
      • Nemotron H and Nemotron H Puzzle
      • Qwen3-Next, Qwen3.5, and Qwen3.8
      • Gemma 3 and Gemma 4
    • KV cache manager V2 is the recommended architecture, with improved scalability and stability. All new models will use V2 by default, existing models will be migrated to V2 gradually, and V1 will be deprecated.

• Known Issues

  • Disaggregated serving may hang during startup or shutdown and leave workers running; affected workloads include GPT-OSS startup and tensor-parallel Llama cleanup.
  • Disaggregated serving with optional execution modes may fail to start.
  • Generation may hang indefinitely across multiple models, including Gemma 4.
  • Repeated create, generate, and shutdown cycles with overlap scheduling may hang across sampler and block-reuse configurations.
  • KV cache manager V2 scheduler workloads may intermittently crash or run out of memory on B200 GPUs.
  • DeepSeek-V3-Lite disaggregated attention data parallelism with overlap may crash; configurations additionally using MTP, CUDA Graphs, and KV cache manager V2 may hang.
  • DeepSeek-R1 V2 FP4 MTP disaggregated serving may crash during generation, and two-node executor initialization may hang indefinitely.
  • GPT-OSS EAGLE serving may deadlock during distributed startup or suffer a worker crash that causes subsequent requests to fail.
  • RocketKV generation may hang with either the TensorRT-LLM or vanilla model path.
  • Disaggregated KV cache transfers may hang after a sender aborts; LoRA generation with chunked prefill may livelock.
  • Qwen3 FP8 LoRA generation may hang.
  • Distributed workload data-parallel inference may crash or hang during shutdown after successful generation.
  • Multi-GPU fused-MoE startup may lose a rank and then hang.
  • Reusing LLM workers for MegaMoE workloads such as DeepSeek V4 Pro may exhaust GPU memory.
  • GPT-OSS dummy-load and W4 configurations may run out of memory while creating the KV cache.
  • DeepSeek-V3.2 FP8 executor creation may run out of memory on H200 GPUs.
  • BF16 LTX-2 and multi-GPU Wan 2.2 visual-generation workloads may produce degraded output quality.
  • Wan text-image-to-video generation with fused normalization may lose conditioning behavior when timestep shapes change.
  • DeepSeek-V3-Lite Helix context-parallel disaggregated serving may produce accuracy loss.
  • Nano V3 Omni FP8 encoder execution with CUDA Graphs may produce accuracy loss.
  • GLM 4.5 NVFP4 disaggregated NIXL serving with KV cache manager V1 may complete with zero accuracy.
  • Qwen2 and Exaone MoE GQA/SWA inference may intermittently produce NaNs and accuracy loss.
  • Qwen3 skip-softmax configurations may produce NaNs and crash with a CUDA device assertion during sampling.
  • Speculative decoding with chunked prefill may exhibit incorrect cache behavior and reduced token acceptance.
  • Stock EXAONE 4.5 and VILA checkpoints may fail to load because their configurations are not registered.
  • Nemotron-Hybrid checkpoints whose hybrid pattern contains a hyphen may fail during configuration loading.
  • DeepSeek-R1 long-context requests with V2 draft KV caching are unsupported when the required sequence length exceeds the initial cache size.
  • Mistral Large 3 FP8 DeepGEMM and NVFP4 EAGLE MoE workloads may crash during startup or generation.
  • LTX-2 NVFP4 generation may fail during startup.
  • Wan pipeline-parallel or Ulysses execution may fail during startup on B200 GPUs.
  • MiniMax-M3 MXFP8 model construction is unsupported.
  • Kimi K3 generation-only workloads with 96-head MLA configurations are unsupported.
  • AutoDeploy standalone generated installations may fail to run when dependency resolution selects incompatible versions.

• Model Support

  • Add Cosmos3-Edge support for text-to-image, text-to-video, and image-to-video generation #16773.
  • Add HunyuanVideo 1.5 text-to-video support #15562.
  • Add GLM-Image text-to-image support #16018.
  • Add Cosmos3 Transfer control-video conditioning #16394.
  • Add multimodal Kimi K3 support #17050.
  • Enable mixed-modality support for Nemotron Nano Omni V3 #16764.
  • Expand Kimi K3 with SiTU MegaMoE and NVFP4 support #17624, #17865, and #17940.
  • Expand Kimi K3 with selectable MLA decode backends, KDA tensor parallelism, MLA decode context parallelism, and SA speculative decoding in disaggregated serving #17800, #17796, and #17939.
  • Expand Qwen3.5 and Qwen3.8 with MoE, attention data parallelism, GDN replay, and improved weight loading #17700.
  • Enable DeepSeek-V4 and DSA models, including DeepSeek-V3.2 and GLM, on SM120 through FlashInfer sparse MLA #16224.
  • Qualify dense Qwen2 models for ModelExpress #16974.
  • Add configuration-driven sqrt-softplus router scoring for Laguna models #17918.

• API

  • Load public TensorRT-LLM exports lazily while preserving top-level names #17281.
  • Deprecate the legacy _autodeploy backend #15642.
  • Remove model_path from TriAttentionKvCacheCompressionConfig and derive the model configuration from the executor #17501.
  • Complete TorchSampler beam-search support and add occurrence penalties #16620 and #17189.
  • Retire two-model speculative-decoding execution in favor of one-model fallback #17366 and #17562.
  • Add prefill CUDA Graph backend and capture-bucket APIs, including experimental breakable CUDA Graphs, while deprecating corresponding TorchCompileConfig fields #16609.
  • Honor SamplingParams.seed and occurrence penalties during one-model speculative decoding #17599 and #17701.
  • Remove conversation IDs from disaggregated request parameters #17618.
  • Add race-free dynamic port allocation through --port 0 and address reporting through --report_addr #17460.
  • Add the /v1/images/edits endpoint for visual-generation models #16987.
  • Expose model-startup and weight-loading metrics through the LLM API and serving stack #15993.
  • Add RuntimeLoRAConfig and startup-time LoRA fusion for VisualGen #17435.
  • Add KV cache manager V2 cold-page codecs while changing pool_ratio ordering and cold-tier statistics #17512.
  • Remove all WIDEEP files #17797.
  • Remove Star Attention and its configuration surface #18025.
  • Revise the VisualGen serving API with path responses, new job states, and /v1/videos/sync #17490.
  • Honor named functions in tool_choice for non-Harmony models #14290.

• Feature

  • Enable KV cache manager V2 by default for GPT-OSS, Gemma 3, and Gemma 4 #16942 and #17396.
  • Extend KV cache manager V2 with DSA support, zero-copy token passing, distributed pool rebalancing, and Helix decode context parallelism #16060, #17308, #17391, and #17866.
  • Add prefix-attention and request-level metrics to KV cache manager V2 #17448 and #17815.
  • Expand disaggregated serving with Python V2 transceivers, Ray-based NIXL transfers, masked DSA cache transfers, decode context-parallel transfers, and enhanced Mamba page tables #16787, #17295, #17283, #17794, and #17691.
  • Add end-to-end dense FP8 LoRA, Blackwell FP8 support, base-model computation overlap, and specialized CUDA Graphs #16810, #17521, #16951, and #17412.
  • Batch TorchSampler beam-search finish-reason reduction #17494.
  • Eliminate redundant one-model sampling-parameter refills #17544.
  • Extend and accelerate DFlash and DSpark with RoPE, sliding-window attention, TensorRT-LLM Gen attention, fused attention/RMSNorm/RoPE, and optimized draft forwarding #16914, #17307, and #17935.
  • Support structured output with DSpark speculative drafting #16965.
  • Support loading MTP weights from standalone checkpoints #17378.
  • Hard-kill all ranks when any executor loop crashes #16592.
  • Enforce multimodal encoder runtime budgets with budgeted output storage #16051.
  • Support isolated out-of-tree builds and incremental artifact copy-back #17524, #17525, and #17538.
  • Optimize Kimi K3 by fusing KDA projections and routing quantization, preallocating residual snapshots, removing MLA padding, and eliminating recurrent-state copies #17311, #17053, #17059, #17684, and #17887.
  • Accelerate VisualGen with fused Wan upsampling, optimized parallel-VAE halo exchange, fused MLP and AdaLN operations, and native Wan VAE reuse for Cosmos3 #17003, #17001, #17693, #17695, and #17555.
  • Enable FlashAttention 4 and parallel VAE in LTX-2 examples #17610.
  • Accelerate DeepSeek sparse attention with a fused MLA prologue and overlapped DSA heuristic write-back #17273 and #16666.
  • Optimize MiniMax-M3 sparse attention with paged KV access, fewer transfers and synchronizations, FP8 indexer QK, improved block selection, decode kernels, and reduced idle time #16857, #17840, #17841, #17236, #17318, #17986, and #17842.
  • Enable INT8 weight-only MoE for non-gated activations #15550.
  • Optimize MoE execution with fused SM90 FP8 block-scale quantization and split routing kernels #16849 and #17893.
  • Add a CFT counted-write path for MoE all-to-all communication #17647.
  • Add tiered and self-sampling CuTe DSL GVR top-K decode kernels #16877 and #17821.
  • Move AllReduce autotuner dispatch into C++ #16902.
  • Eliminate quadratic copying during KV cache block reuse #17848.
  • Add low-M BF16 GEMM dispatch for SM10x decoding #17889.
  • Improve small-BF16-context performance with targeted fallback selection #18196.
  • Reduce runtime overhead from request-context construction, response timing, hang detection, and profiling bookkeeping #17582, #18053, and #17531.
  • Cache Whisper suppress-token indices on the GPU #17583.

• Fix

  • Enforce Responses conversat...

• Known Issues

  • Workloads combining torch.compile with CUDA graphs can fail during initialization or inference with allocator assertions, missing outputs, or illegal memory accesses.
  • MLA models using MTP with multi-stream or piecewise CUDA graphs on SM120 can asynchronously crash with an illegal memory access.
  • Workloads using KVCacheManagerV2 may intermittently report a teardown failure after otherwise successful inference; this is observed with DeepSeek-V4-Flash.
  • Diffusion image and video pipelines can show output-quality regressions when conditioning defaults or fused normalization change numerical behavior; this is observed with Cosmos 3 Nano T2I/T2V and Wan 2.2 T2V.
  • Multi-GPU low-precision MoE models using fused residual normalization and all-reduce can suffer severe accuracy loss; observed examples include Nemotron V3 Ultra NVFP4 and GPT-OSS W4.
  • Qwen3 MoE models may fail during startup when using FP8 block-scale quantization on Blackwell.
  • Mistral Large-family NVFP4 configurations can show materially reduced GSM8K accuracy, as observed with Mistral Large 3 675B.
  • Multi-rank hybrid models using shared FlashInfer JIT artifacts can fail to start when ranks compile the same kernel concurrently.
  • High-concurrency disaggregated serving over Python NIXL with UCX 1.21 can crash during KV-cache transfer or request cancellation.
  • Hybrid recurrent-attention models using Mamba/GDN state caches can be rejected at startup because required live and dummy slots are underestimated; observed examples include Nemotron V3 Ultra NVFP4 and Qwen3-Next BF16.

• Model Support

  • Add Kimi K3 with KDA kernels, optimized MoE, parsers, speculative decoding, and disaggregated serving (#17054, #17190, #17225, #17269, #17327, #17334)
  • Add MiniCPM-V 4.6 image and video support to the PyTorch backend (#15976)
  • Add Whisper support to the PyTorch backend (#16141)
  • Add Qwen-Image-Layered and Qwen-Image-Edit-2511 pipelines with CFG parallelism and TeaCache/Cache-DiT (#15096, #16095, #16384, #16339)
  • Add distilled Cosmos3 four-step text-to-image, image-to-video, and video-to-video generation (#16563, #16690, #16155)
  • Add the FastWan2.2 TI2V-5B three-step text-to-video pipeline (#16162)
  • Enable reference-image conditioning for FLUX.2 (#16644)
  • Enable dual-topology parallelism for LTX-2 two-stage generation (#16502)
  • Support mixed image, video, and audio inputs for Qwen2-VL and Qwen3-VL (#16337)
  • Enable multimodal encoder caching, side-stream integration, and partial cache hits for Qwen3.x and Gemma4 (#16817, #16662, #16554, #17231)
  • Add Gemma4 MTP assistant support (#15833)
  • Expand Qwen3, Qwen3.5, and Qwen3-Next compatibility with Qwen3.5-397B BF16/FP8 refit, safer loading, MoE/GDN/MTP fixes, and W4A8 execution (#16642, #16936, #17051, #17120, #17196)
  • Improve Mistral checkpoint, tokenizer, chat, and multimodal format handling (#15123)
  • Improve Nemotron-H quantized loading (#16833)
  • Handle EXAONE 4.5 33B memory constraints (#16992)

• API

  • Expand TorchSampler with repetition, frequency, and presence penalties, min-p, no-repeat n-grams, and per-request seeds; reject NaN sampling values and raise top_logprobs to 100 (#16485, #16590, #16594, #17172, #17159, #16851)
  • Honor compatible generation_config.json sampling defaults through generation_config="auto" and --generation-config auto (#17213)
  • Add per-request priority to OpenAI chat and completion APIs (#15329)
  • Improve Qwen3 tool-call and DeepSeek-R1 reasoning-parser completion handling (#16866, #17157)
  • Configure disaggregated-serving HTTP keep-alive with server_keep_alive_timeout (#16430)
  • Protect internal disaggregated metadata with shared-secret request signing (BREAKING) (#16002)
  • Refactor multiprocess per-request metrics with header/SSE responses and JSONL output; deprecate perf_metrics_max_requests (BREAKING) (#16668)
  • Replace Mamba cache management with KVCacheManagerV2-backed snapshots and move controls under mamba_state_config (BREAKING) (#16598)
  • Move kv_cache_config.block_reuse_policy to block_reuse_config.policy and add max_num_turns (BREAKING) (#16883, #17277)
  • Add TriAttentionKvCacheCompressionConfig for TriAttention KV-cache compression (#16957)
  • Add advanced_sampling_mode controls for one-model MTP sampling (#16561)
  • Expose prototype DSA MTP indexer top-k sharing through index_share_for_mtp_iteration (#15806, #17149)
  • Enable encoder and mixed encoder-decoder CUDA graphs through new LLM arguments (#16706)
  • Add fp8_context_mla_kv_len_cap for context-MLA workspace and KV-capacity tuning (#16399)
  • Add the MEGAMOE_CUTEDSL backend option and disable deprecated WIDEEP selection (#16190, #17119)

• Feature

  • Introduce C++ KVCacheManagerV2 with paged Vanilla attention and FlashInfer block reuse (#14047, #16714, #17106)
  • Add batched physical compaction for KV-cache compression (#16836)
  • Enable VisualGen tensor parallelism with Attn2D (#16677)
  • Accelerate VisualGen with FLUX.2 fused NVFP4 SwiGLU, Wan layernorm/shift-scale/quant fusion and temporal VAE batching, and Qwen-Image fast paths (#16143, #15762, #17002, #16142)
  • Enable Marlin NVFP4 on Ada Lovelace and W4A16 NVFP4 on SM120 (#16749, #16511)
  • Expand Blackwell kernels with CuTe DSL FP8/FP16 MLA decode attention and optimized fused MHC half-MMA (#15138, #16799)
  • Optimize sparse-attention top-k and indexing with CuTe DSL radix/GVR kernels, selective DSA K-cache allocation, and reduced prefill work (#15756, #16457, #16558, #16789)
  • Optimize MiniMax M3 with fused projections, MoE routing, QK-norm/RoPE/quantization, SwiGLU and AllReduce-residual-RMSNorm, leaner Index-K caching, and deduplicated eager plans (#16904, #16859, #16906, #17093, #16905, [#17091]...

• Known Issues

  • Deepseek-V4-Pro can hang on GB300 disagg setups.
  • DeepSeek-R1 NVFP4 multi-GPU (PP4 + MTP) can crash with an MPI worker exit during execution (seen on GB300).
  • DeepSeek-V3-Lite BF16 + chunked prefill (Python scheduler) can hang / fail to complete.
  • Qwen MoE multi-LoRA (routed experts, varying ranks, eager) can fail during sampling.
  • Gemma3-1B FP8 prequantized with torch.compile can fail during CUDA graph capture (PyTorch CUDA allocator internal assert).
  • Qwen3.5-35B-A3B BF16 (CUTLASS, TP1 on A100) can hit CUDA OOM.

• Model Support

  • Load DeepSeek V4 mixed-precision NVFP4 checkpoints (#16433)
  • Enable W4A8 checkpoint loading for Gemma4 K=V layers (#16797)

• VisualGen

  • Implement uneven TP linear for VisualGen models (#14875)

• API

  • Support multi-process HTTP frontends on the classic IPC executor path in trtllm-serve (#16523)
  • Emit initial KV cache stats at startup for external metric scrapers (#12596)

• Feature

  • Default GPT-OSS and GLM-5 to the Python KV-cache transceiver (#16479, #16524)
  • Add MiniMax-M3 MSA sparse attention backend and migrate MiniMax M3 to loader v2 for TP8 support (#16291, #16468)
  • Support MiniMax M3 for disaggregated serving (#16017)
  • Support top_p_decay in the PyTorch TorchSampler (#16184)
  • Support GDN MTP replay (#16464)
  • Support Nemotron dynamic-tree MTP decoding (#15582)
  • Support rejection sampling under attention DP (including LM-head TP) (#16544)
  • Support MARLIN MoE with MTP and attention DP + EP (#16597)
  • Support fine-grained context chunk management (#16427)
  • Add TensorRT-LLM runtime integration for KV cache compression (#15697)
  • Add per-conversation KV cache block reuse (#16115)
  • Improve PyTorch encoder-decoder support and performance (#16158)
  • Optimize fused QKNormRoPE kernel (#16633)

• Fix

  • Account for resume utilization in KV constraints and fix KV cache estimation capacity (#16484, #16545)
  • Make auto host-tier sizing rank-aware in KVCacheManagerV2 (#16106)
  • Fix DSpark all-reduce of draft MoE output under non-attention-DP TP (#16617)
  • Fix DSpark rolling-window slot collision in disaggregated serving (#16772)
  • Move Qwen-VL mRoPE seq-slot delta cache correctly (#16537)
  • Avoid attention workspace resize during CUDA graph capture (#16319)
  • Make FlashInfer sampling op wrappers opaque to Dynamo (#16732)
  • Allow trtllm-gen MoE autotuner when local_num_experts < top_k (#16653)
  • Use attention cache dtype for disaggregated transfer (#16505)
  • Fall back to disagg_request_id in Python NIXL decode receiver (#16720)
  • Fix MPI world heap corruption during teardown (#16550)
  • Stop thinking-budget processor from re-forcing the reasoning end tag (#16785)
  • Fix GPT-OSS router token identity (#16760)
  • Exclude ADP/CUDA-graph dummy requests from speculative-decode acceptance stats (#16571)
  • Use explicit errors instead of asserts for real-dataset input (#16257)

• Documentation

  • Remove legacy Python relics and refresh docs after the TensorRT backend removal (#16612)

• Benchmark

  • Support DeepSeek-V4 in layer-wise benchmarks (#16774)

• Test & Infra

  • Assert MTP acceptance length in ADP + LM-head-TP accuracy tests (#16521)
  • Add DeepSeek R1/V3.2/V3-Lite disaggregated accuracy tests (#16344)
  • Update disaggregated GB200 test configs (#16481)
  • Raise BF16 Laguna DFlash memory skip threshold to prevent OOM (#16627)
  • Remove MiniMax-M2 TP16 multinode eval test case (#16542)
  • Add DeepSeek V4 Pro QA cases and DeepSeek-V4 B200/B300 single-node GPU tests (#16611, #16661)
  • Migrate Kimi disagg tests to Transceiver v2 and trim tests (#16482)
  • Split slow B300 attention unit tests (#16709)
  • Remove legacy TensorRT backend tests, examples, and CI plumbing (#16610)
  • Remove --trt_root and stop installing the TensorRT SDK into images (#16608)
  • Use main as default target branch in L0_MergeRequest_PR (#16480)
  • Preserve pytest progress with S3 capture (#16268)
  • Add support to run NGC container scanning in pre-merge (#15602)
  • Add per-stage option to cap or disable stage-level infra retries (#16551)
  • Require approval for PerfSanity wildcard runs (#16777)
  • Surface SLURM device faults to the failure classifier (#16557)
  • Update flashinfer-python from 0.6.14 to 0.6.15 (#16530)

What's Changed

• [https://nvbugs/6245862][fix] Fix MoE EP divisibility check for EPLB configurations by @JacobHu-NV in #15410
• [TRTLLMINF-127][infra] Upgrade dependencies for NGC PyTorch 26.05 stack by @EmmaQiaoCh in #15087
• [None][test] Waive 7 failed cases for main in QA CI by @trtllm-agent in #16042
• [None][feat] Add Laguna DFlash drafter support by @joerowell in #15666
• [None][infra] Enable B300 stages by @yiqingy0 in #16359
• [https://nvbugs/6428144][test] Unwaive GB300 DeepSeek-R1 gen-only disagg PerfSanity by @chienchunhung in #16267
• [None][chore] Remove disagg-devs co-ownership of mla.py in CODEOWNERS by @Tabrizian in #16385
• [TRTLLM-12721][fix] Add gated C++ NIXL in-flight cancellation and safe cleanup by @chienchunhung in #15238
• [https://nvbugs/6374873][fix] Allow fp4 KV Cache + non-FP4 Mamba State by @VALLIS-NERIA in #16304
• [https://nvbugs/6435642][fix] detect killed MPI executor workers by @chienchunhung in #16338
• [None][test] Fix the multinode test case on DGX-Spark(perf skipped and func hang) by @JennyLiu-nv in #16209
• [TRTLLM-14024][feat] Prune CuTe DSL NVFP4 GEMM autotuner tactics with nvMatmulHeuristics by @peaceh-nv in #15909
• [TRTLLM-12373][feat] RMSNorm nvfp4 quant fusion for DS V3.2 / Kimi-K2.5 by @JunyiXu-nv in #14848
• [None][test] Refine CBTS logic for handling -Perf- stages by @yufeiwu-nv in #16328
• [None][infra] Assign KV cache manager v2 test ownership by @jiaganc in #16372
• [None][fix] fix missing torch_dtype_to_binding by @bo-nv in #16401
• [https://nvbugs/6426860][fix] Stabilize compressor BF16 tolerance by @mingyangHao in #16376
• [https://nvbugs/6403909][test] Remove attention backend test waiver by @yuxianq in #16400
• [None][chore] Rename AutoDeploy standalone package to Paragraf by @bmarimuthu-nv in #16324
• [None][test] Split test_moe_backend TRTLLM by quant=None to avoid in-process IMA cascade by @leslie-fang25 in #16068
• [None][fix] Fix Gemma4 illegal memory access when max_seq_len is at most the sliding window size by @Hudayday in #16099
• [https://nvbugs/6396422][fix] Disable kv-cache reuse for MiniMax-M2 by @tcherckez-nvidia in #16383
• [None][test] Waive 1 failed cases for main in QA CI by @trtllm-agent in #16412
• [None][fix] align KV slice token_range.end with transferred block count by @chuangz0 in #15824
• [None][feat] Python transceiver support cpp cache manager + offload by @chuangz0 in #15245
• [None][chore] add disagg-devs as code owner for disaggregated tests by @Shixiaowei02 in #16415
• [TRTLLM-13212][refactor] Unify sampler ops and clean up dead code in TorchSampler by @zhaoyangwang-nvidia in #16365
• [https://nvbugs/6260897][fix] Relax line-589 assertion to accept either IN_PROGRESS or SUCCESS (FAILURE… by @tensorrt-cicd in #15121
• [None][infra] Waive 2 failed cases for main in pre-merge 47854 by @trtllm-agent in #16423
• [https://nvbugs/6445332][test] Remove test waivers for nvbug 6445332 (code fix merged in #16326) by @trtllm-agent in #16316
• [None][infra] Tail Slurm job logs when job is no longer active. by @mzweilz in #15455
• [None][chore] KVCacheManagerV2: Python and test preparation for a C++ backend by @lowsfer in #16218
• [https://nvbugs/6283537][fix] Unwaive TestQwen3_5_4B::test_bf16 by @amukkara in #16392
• [None][infra] Waive 1 failed cases for main in pre-merge 47597 by @trtllm-agent in #16361
• [None][infra] Waive 1 failed cases for main in pre-merge 47847 by @trtllm-agent in #16443
• [None][infra] Waive 1 failed cases for main in pre-merge 47847 by @trtllm-agent in #16442
• [https://nvbugs/6269388][fix] Raise error to enforce HMAC encryption in IPC by @yibinl-nvidia in #16249
• [None][infra] Waive 1 failed cases for main in pre-merge 47953 by @trtllm-agent in #16450
• [None][infra] Waive 1 failed cases for main in pre-merge 47930 by @trtllm-agent in #16452
• [TRTLLM-12352][feat] add post-transform capability profi...