• TransformerEngine upgraded from 2.17 to 2.18
• Added support for FP4 in 1F1B A2A overlap (#6135)

MoE

• Quantile Balancing router: adds auxiliary-loss-free load balancing using per-expert routing biases derived from quantile estimates (#5349).
• Fused shared-expert MLP: enables shared experts through grouped GEMM, providing the path for fused GEMM, SwiGLU, and quantized execution (#5604).
• Packed-sequence MoE dispatch: adds THD sequence-packing support to the DeepEP and HybridEP dispatcher backends (#5008).
• MoE routing analysis: adds training/inference routing traces plus expert-concentration and one-layer-ahead predictability analysis tools (#5220).

Model Architecture and Support

• HybridModel: next-gen model class.
  • GPTModel to HybridModel guide published.
  • Added ability to directly load GPTModel checkpoints (#5792).
  • Part 1 of the design doc is complete, part 2 will be completed for 0.20.0.
  • Deepseek v4 migrated to HybridModel on the dev branch (#5042, #5485, #4949).
• GPTModel: deprecation announcement.
• DeepSeek Sparse Attention (DSA): adds backend-neutral CP and THD support, pipeline-aware indexer-loss scaling, IndexShare, and fused cuDNN and TileLang implementations (#5244, #5246, #5099, #5049).
• Multi-Latent Attention in HybridModel: adds MLA layer support, Q/K norm spec resolution, and fused MLA projection handling to HybridModel (#4452).

Parallelism

• FSDP: MFSDP v2 is under development. See the design doc for links to the project tracker and implementation code.
• NCCL EP support: adds NCCL EP token dispatcher backend (#5129, #5735).
• Fuse per-sequence AlltoAll: creates a unified AlltoAll (#4913).

Performance and Memory

• CUDA Graph-compatible activation offloading: enables fine-grained activation offloading during capture and replay, including deferred transfers and fused grouped-MLP offloading (#3697, #5082).
• Streaming quantized checkpoint loading: dequantizes FP8, MXFP8, blockwise-FP8, and NVFP4 tensors incrementally instead of allocating BF16 scratch space for the entire model (#4451).
• Partial CUDA Graph memory optimization: moves graph surfaces into the global CUDA Graph memory pool and removes references and copies that retained excess buffers (#5451).

Precision and Optimizers

• MXFP8 training paths: take a look at our Nemotron 3 Super MXFP8 example
• Lion distributed-optimizer support: routes Lion through DistributedOptimizer and adds single-moment checkpoint support, including Muon configurations that use Lion for scalar parameters (#5742).

Inference

• Disaggregated-inference foundations: adds heterogeneous inference-shard specifications, multi-destination refit, KV/Mamba-state resharding, local deterministic planning, and a NIXL copy backend (#5186, #5187, #5188, #5872).
• Full-model CUDA Graphs for MTP inference: enables full-model graph execution for Multi-Token Prediction inference (#4950).
• Load-aware prefix-cache routing: routes requsts across inference workers using prefix-cache load information (#5607).

Datasets

• Inter-document attention masking: GPTDataset now emits document-boundary metadata, resets positions per document, and uses THD attention to prevent cross-document attention within packed samples, including TP and CP support (#5298).

Multimodal

• MIMO heterogeneous training: expands the current MIMO bullet with end-to-end non-colocated execution, stock training-loop integration, encoder prefetch, DDP overlap, and Nemotron MoE VLM/RADIO examples (#5602, #5833, #5979).

Known Issues

• TE fused cross-entropy may degrade stability: setting cross_entropy_fusion_impl to te causes convergence issues on some models. Use native instead, or upgrade Transformer Engine to 2.18, which carries the fix (TE PR #3193).
• SPCX grouped all-to-all payload corruption at high expert parallelism: in the affected NCCL 2.30.5/SPCX stack, stale or uninitialized ncclIbRequest.send.onlyWriteImm state can misclassify sends in grouped variable-split BF16 all-to-all operations, causing payload loss or corruption. MCore's MoE alltoall dispatcher can expose this issue at high EP configurations, but the fault originates in the SPCX transport rather than the model or router. As a workaround, disable the external NCCL network plugin and use NCCL's internal IB transport (NCCL_NET_PLUGIN=none NCCL_NET=IB), or use the HybridEP dispatcher (Megatron-Bridge issue #5462).

Parallelism

• A2A overlap for Megatron-FSDP - Bridge Megatron-FSDP's hook-based lifecycle with the EP overlap schedule's direct sub-module execution path (#3797)
• Packed sequence support for GDN - Packed sequence support for gated delta net on main, following the earlier implementation work (#2645, #2644)
• Permute fusion into Hybrid EP - Fuse permute / unpermute with dispatch / combine into a single kernel path for Hybrid EP (#4089)

MoE

• sqrtsoftplus MoE score function: Add a new score function sqrtsoftplus to the router (#3673)
• Distributed Layerwise Optimizer: Adds a shard-aligned parameter layout for LayerWiseDistributedOptimizer that guarantees no parameter is split across shard boundaries (#4509)
• CUDA Graph Interface Refactor: decomposes the overloaded cuda_graph_scope field into three dedicated, semantically distinct concepts and cleans up naming throughout (#4292)

Performance and Memory

• Batch-processing utilities and PP for SFT - Consolidate get_batch utilities and enable pipeline parallel SFT in THD paths (#4103)
• Emerging optimizer refactor - Refactor optimizer infrastructure to generalize beyond Muon and support additional emerging optimizers (#4113, #4119)
• Chunked MLP training - Enable chunked MLP during training, extending prior inference-prefill support (#3656)
• GroupGEMM + SwiGLU + quantize fused MLP - Transformer Engine fused grouped MLP support, grouped quantized tensor plumbing, checkpoint compatibility, op fuser work, and related MLP infrastructure (#4636)
• Full-model Cuda Graph with paged stashing - Decouple oversized compute buffers from properly sized activation buffers for backward-pass storage, continuing the CUDA Graph / dropless MoE memory work (#4247)

Precision

• MXFP8 improvements and fixes - Fine-grained param gather configuration, forced param all-gather eval fixes, numerical fixes when DP overlap is disabled, and follow-through on MXFP8 FP8-param-gather support (#2582, #4181, #4562, #4800)
• NVFP4 param gather - Make FP4 param gather work correctly with mixed precision in the NVFP4 recipe (#4358)

Inference

• New dispatcher for MoE inference - AllGatherV dispatcher for inference and simplification of the previous dispatcher path (#4258)
• vLLM grouped GEMM backend - Port vLLM grouped GEMM to the inference-optimized MoE backend (#4566)
• CUDA Graphs for MTP inference - Enable CUDA Graphs for MTP inference (#4260)

Model Architecture and Support

• Flextron - Post-training method for converting a single parent LLM into a nested family of submodels at different parameter budgets (#4429)

Multimodal

• MIMO core primitive - Add the core primitive for heterogeneous TP / DP MIMO training, including colocated bridge communication. This is a step toward the broader MIMO goal of early-fusion multimodal architectures with modular vision, audio, and video encoders, independent nD parallelism per module, and colocated or non-colocated training layouts (#1375, #4368)
• A2A overlap for Megatron-FSDP - Bridge Megatron-FSDP's hook-based lifecycle with the EP overlap schedule's direct sub-module execution path (#3797)
• Packed sequence support for GDN - Packed sequence support for gated delta net on main, following the earlier implementation work (#2645)
• Permute fusion into Hybrid EP - Fuse permute / unpermute with dispatch / combine into a single kernel path for Hybrid EP (#4089)
• Distributed Layerwise Optimizer: Adds a shard-aligned parameter layout for LayerWiseDistributedOptimizer that guarantees no parameter is split across shard boundaries (#4509)
• CUDA Graph Interface Refactor: decomposes the overloaded cuda_graph_scope field into three dedicated, semantically distinct concepts and cleans up naming throughout (#4292)
• Batch-processing utilities and PP for SFT - Consolidate get_batch utilities and enable pipeline parallel SFT in THD paths (#4103)
• Emerging optimizer refactor - Refactor optimizer infrastructure to generalize beyond Muon and support additional emerging optimizers (#4113, #4119)
• Chunked MLP training - Enable chunked MLP during training, extending prior inference-prefill support (#3656).
• GroupGEMM + SwiGLU + quantize fused MLP - Transformer Engine fused grouped MLP support, grouped quantized tensor plumbing, checkpoint compatibility, op fuser work, and related MLP infrastructure (#4636)
• Full-model Cuda Graph with paged stashing - Decouple oversized compute buffers from properly sized activation buffers for backward-pass storage, continuing the CUDA Graph / dropless MoE memory work (#4247)

Known Regression

We have found a performance regression of ~10% in our T5 performance tests on H100s. This does not affect GB200s. We have root caused this regression to the base PyTorch image upgrade from 26.02 to 26.04. We are actively investigating and looking to fix this regression soon.