NeMo AutoModel 26.10 Release Notes

Highlights

• Expanded model coverage. New training paths for DeepSeek-V4.1-Flash,
  GLM-5.3-Flash, Qwen3.8-Flash-Next, MiMo-V2.5-Pro and MiMo-V2.6, and additional
  vision-language models.
• More MoE training options. Mixture-of-Kittens (MoK) BF16 and MXFP8 experts,
  compact HybridEP routing, and MXFP4-resident DeepSeek-V4-Flash experts for LoRA.
• Speculative and discrete-diffusion training. DFlash 2, additional Kimi K3
  and DeepSeek-V4.1 draft-training paths, streaming EAGLE-3 feature components,
  I-DLM training, and SCDD training and sampling.
• Text-to-image fine-tuning. HunyuanImage-3.0 with expert parallelism and
  Qwen-Image-2.1 with full-parameter and LoRA recipes.
• Training and integration improvements. Native FlashAttention-4, explicit
  packing contracts, weighted domain mixtures, lower-memory checkpoint loading,
  and public registration of custom model architectures.

New and Expanded Model Support

LLM and MoE

• DeepSeek-V4.1-Flash: Native model and checkpoint conversion, attention and
  expert LoRA, packed SFT and context parallelism, H100-80GB recipes, and MedPix
  vision SFT. The release includes the CP8 gradient-synchronization fix.
  (model guide, #3855, #3872, #3879, #3962, #4221)
• GLM-5.3-Flash: Native hybrid KDA/DSA model, vision processing, distributed
  checkpoint loading, and packed context-parallel training. The MedPix recipe
  uses EP72/CP2 on 72 H100 GPUs; sparse attention combines FlashMLA forward with
  cuDNN backward. TP, PP, video training, and full-model single-GPU loading are
  outside this model's supported path. GLM-5.2 and GLM-5.3 also gain cuDNN DSA
  training backends.
  (model guide, #3699, #3127, #3747)
• Qwen3.8: Adds 27B fine-tuning recipes and native Flash-Next language-only
  training with a sharded Engram embedding table, sparse QSA attention, packed
  sequences, and contiguous context parallelism. Flash-Next's QSA indexers remain
  frozen; its checkpoint MTP head is not loaded. TP and PP are disabled in the
  documented Flash-Next configurations.
  (model guide, #3560, #3690)
• MiMo: Adds MiMo-V2.5-Pro, MiMo-V2.6, and MiMo-V2.6-Pro support. The
  MiMo-V2.6-Pro-RL recipes cover packed 64K Tulu3 text, MedPix full fine-tuning,
  and MedPix LoRA on GB200. Keep expert-parallel groups within an NVLink domain.
  (recipe guide, #2514, #3992, #4054)
• Additional architectures: Dense Nemotron-H Nano 4B, a Laguna XS 2.1
  training recipe, and encoder-decoder fine-tuning through
  NeMoAutoModelForSeq2SeqLM, including a T5-small SQuAD example. The seq2seq
  path does not wire up CP, TP, PP, or sequence packing.
  (#2670, #3639, #2860)

Vision-Language Models

• DeepSeek-V4-Flash Vision and Muse Glimmer: Adds the DeepSeek-V4 vision
  path and native Muse Glimmer training, checkpoint conversion, and distributed
  model support.
  (#3772, #3476)
• Cohere North Micro Vision: Adds Shopify product-catalogue and RDR LoRA
  recipes, including checkpoint resume. This preview model uses remote-code
  dependencies and is excluded from automatic release-scope CI discovery.
  (#3518, #3400)
• Existing VLM improvements: Gemma 4 gains D512 FFPA attention on SM100 and
  sliding-window communication improvements. MiniMax-M3 gains SM100 sparse
  attention with a custom backward pass. Qwen3.5 gains packed vision training
  with Transformer Engine and MTP with VLM context parallelism.
  (#3742, #2984, #3809, #4303, #3990)

Speculative Decoding and Discrete-Diffusion LLMs

• DFlash 2: Draft model, trainer, Qwen3 and Qwen3.8-27B recipes, dynamic
  convolution, and candidate-path selection. Sampling supports top-p and top-k.
  (#3605, #3627)
• Additional draft targets and parallelism: Kimi K3 DSpark, EAGLE-3, and
  DFlash training; DeepSeek-V4.1 DSpark training; and Ulysses context parallelism
  for the EAGLE-3 draft.
  (#3263, #3286, #3287, #3875, #3356)
• Streaming supervision components: EAGLE-3 feature production and loading,
  an asynchronous producer, and a safetensors-backed shared-directory feature
  store with backpressure.
  (#3085, #3094)
• I-DLM and SCDD: I-DLM all-masked training uses dllm.mode: idlm; SCDD
  adds mixed mask/uniform corruption, training, and sampling with
  dllm.mode: scdd. Neither new mode supports context parallelism. SCDD does
  not currently maintain the reference method's EMA weights.
  (#2437, #3354)

Diffusion

• HunyuanImage-3.0: Native MoE text-to-image implementation with FSDP2,
  expert parallelism, LoRA, full fine-tuning, preprocessing, and generation.
  Recipes cover single-node LoRA and four-node full fine-tuning. TP, PP, CP,
  image-to-image conditioning, and text-generation training are unsupported.
  (model and recipe guide, #4113, #4129)
• Qwen-Image-2.1: Full-parameter and LoRA text-to-image fine-tuning, a
  dedicated latent/prompt preprocessing path, and generation recipes. Requires
  a Diffusers build providing QwenImage21Pipeline.
  (model and recipe guide, #4027)
• Validation: Periodic diffusion validation loss, retention of validation
  tail batches, and sample-weighted validation loss.
  (#3579, #4291)

Retrieval and Embeddings

• Context-aware reranking: Train rerankers with query-context pairs and
  configurable prompt augmentation.
  (#3786)
• Mistral3 vision-language retrieval: Embedding training, reranking, and
  multimodal hard-negative mining for text, images, and image-plus-text
  documents. Exports preserve retrieval prompts, pooling, normalization,
  attention policy, and scoring temperature; release backports fix portable
  reranker exports and Pixtral image-attention boundaries.
  (#3915, #4197, #4267)

Training, Parallelism, and Performance

• MoK experts: Optional dispatcher: mok integration for fused dispatch
  and expert computation, plus MXFP8 experts. Supported shapes, precision,
  topology, and adapter combinations are checked explicitly.
  (#3422, #3606)
• Lower-memory DeepSeek-V4-Flash LoRA: MXFP4-resident frozen experts avoid
  keeping a full BF16 expert copy. This path requires expert parallelism and
  torch_mm experts with the torch dispatcher; DeepEP experts are unsupported.
  (#2548)
• MoE and Kimi K3 efficiency: Compact HybridEP routing, reuse of DeepEP
  dispatch layouts during activation-checkpoint recomputation, fused gra...

Highlights

• Frontier-scale MoE and multimodal models. Train Moonshot AI's Kimi K3 (2.8T/104B active), Thinking Machines Lab's Inkling (975B/41B active), MiniMax AI's MiniMax-M3 VL (428B/22B active), Poolside's Laguna, and Zhipu AI's GLM-5.2.
• Long-context training. Shard packed multi-document sequences with block-diagonal variable-length context parallelism, and shard vision towers by frame to remove the redundant vision compute of a CP fold; recipes reach 128K tokens at EP8/CP32.
• New attention and kernel backends. Select FlashAttention 3 and 4, MagiAttention, FFPA for headdim=512 layers, QuACK linear/RMSNorm/RoPE kernels, and scoped partial CUDA graphs for MoE.
• Resilient checkpointing. Checkpoint on preemption signals, write to msc:// cloud directories, consolidate async saves in the background, bound checkpoint retention, and resume past an interrupted save.
• Diffusion. Fine-tune and generate with LTX-2.3 joint video+audio and Qwen-Image-Edit-2511, and run context parallelism through the diffusers ContextParallelConfig API.

New Hardware and Precision Support

• FP8 draft training. Extends the SFT recipes' top-level fp8: block to every speculative recipe, swapping the draft's nn.Linear layers to torchao Float8Linear on SM89 and newer, with emulate: true for older GPUs. (#2963)
• Quantized checkpoint ingestion. Dequantizes MXFP4 dequantization for Kimi K3 and MXFP8 dequantization for MiniMax-M3 for BF16 training, keeps Kimi-K2.5-VL LoRA keys out of INT4 quantization on save, and fixes a TileLang boolx8 backward codegen failure on the DeepSeek-V4 path. (#3259, #3295, #3467)

New and Expanded Model Support

LLM and MoE

• Moonshot AI: Kimi K3 (2.8T / 104B-active MoE) — text-only HellaSwag full SFT at EP32/PP8 on 256 GB200 GPUs. (docs, #3259, #3283)
• Zhipu AI: GLM-5.2 (MoE with IndexShare DSA) — HellaSwag and Tulu-3 4K at EP64/PP4, Tulu-3 32K at CP8/EP64/PP4, LoRA at EP128; selectable TileLang DSA kernels. (docs, #2633, #2691, #2695, #3222)
• Poolside: Laguna S 2.1 (118B / ~8B-active MoE) — HellaSwag full SFT at EP16. (docs, #3148)
• DeepSeek: DeepSeek-V4 Flash (MoE) — Tulu-3 SFT at CP8/PP4/EP32 and pretraining at PP4/EP32; packed THD under context parallelism. (docs, #2590, #2731, #3227, #3231)

VLM

• MiniMax AI: MiniMax-M3 VL (428B / 22B-active MoE) — MedPix full SFT at EP32/PP4 on 16 nodes, MedPix LoRA at PP4/EP8, a CP2 comparison recipe, and 16K packed Tulu-3 text SFT at CP8. (docs, #2538, #2551)
• Thinking Machines Lab: Inkling (975B / 41B-active MoE; text, image, video, audio) — MedPix SFT at PP8/EP32 on 32 nodes. (docs, #3095, #3281, #3358)
• Alibaba: Qwen3.5 and Qwen3.6 VLM (122B-A10B MoE, 27B dense) — 128K packed sequences at EP8/CP32 on 16 nodes with a trainable vision tower, dense context parallelism, and matched CP1/CP2 27B MedPix recipes. (docs, #2505, #3186, #3209)
• Google: Gemma 4 (31B and E2B/E4B dense, 26B-A4B MoE) — context parallelism with 16K Tulu-3 at CP8, 64K Tulu-3 at CP16, and 4K MedPix at EP8/CP2; plus fp32 SDPA to avoid NaNs on Hopper and an exact fp32 reference router for MoE parity. (docs, #1914, #2592, #2621, #3141)
• Moonshot AI: Kimi-K2.5 VL (MoE) — LoRA tensors kept out of INT4 expert quantization on save, so adapters reload on resume. (#3295, #3431)
• NVIDIA: Nemotron-Parse — corrected preprocessing stopping the RADIO encoder from normalizing already-normalized images a second time. (#3331)
• MTP on multimodal inputs. Pre-rolls pre-fused inputs_embeds per depth so audio and vision positions keep their continuous embedding instead of being re-embedded from a padding token id. (#2510)
• Pre-extracted video frames. Loads a video field holding a list of image paths directly as a frame sequence with no decoder, padded to the processor's temporal_patch_size. (#3211)

Diffusion

• Lightricks: LTX-2.3 (video + audio) — full-finetuning, LoRA, and generation recipes that train video and audio jointly and mux the generated waveform into the output mp4; --processor ltx2 preprocessing requires an 8n+1 frame count. (docs, #3165, #3372)
• Alibaba: Qwen-Image-Edit-2511 — cached, full-parameter instruction-based editing with an offline latent and prompt cache encoder, an eight-GPU BF16 recipe, and an image-edit preprocessing subcommand. (docs, [#3217](http...

Release Notes

• Highlights

  • Blackwell and FP8 support. Run DeepSeek-V4 with TileLang kernels, train MoE models with MXFP8 on GB200, and use context parallelism for Qwen3.5/3.6 MoE VLM. More details in our performance summary
  • Agent SFT. Train Qwen2.5-3B on multi-turn, function-calling ChatML or ShareGPT data with full-SFT and LoRA recipes. The data adapter preserves tool calls, tool responses, and optional reasoning_content.
  • Speculative decoding. Train EAGLE 1/2/3/3.1, P-EAGLE, and DFlash drafts for Llama, Phi, Qwen3, Qwen3-MoE, GPT-OSS, and Nemotron Nano targets.
  • dLLM. Train DFlash drafts and fine-tune LLaDA2 or Nemotron-Labs-Diffusion with the new discrete-diffusion workflows.
  • Diffusion. Fine-tune and generate with FLUX.2-dev, Qwen-Image, and Wan 2.2 T2V-A14B; update FLUX, Hunyuan, and Wan 2.1 recipes for LoRA and throughput.

• New Hardware and Precision Support

  • GB200 MXFP8 MoE. Adds Transformer Engine MXFP8 grouped-expert training for Qwen3-MoE-30B, GPT-OSS-120B, and Qwen3-MoE-235B, and standardizes the bf16 expert backend on Transformer Engine grouped linear layers.

• New and Expanded Model Support

• LLM and MoE

  • DeepSeek-V4 Flash & Pro. Adds a native model, checkpoint adapter, HellaSwag full-SFT and LoRA recipes, a packed-sequence recipe, and large-scale TileLang PP/EP recipes. The implementation covers hash routing, hyper-connections, sparse/compressed attention, FP4/FP8 checkpoint loading, and multi-token prediction.
  • ERNIE 4.5. Adds HellaSwag recipes for the 0.3B dense and 21B-A3B MoE checkpoints.
  • MiMo-V2-Flash. Adds a HellaSwag fine-tuning recipe and a native model implementation with checkpoint conversion.
  • Ling 2.0. Adds Mini, Flash, and 1T model paths with SFT, LoRA, HellaSwag, SQuAD, and pipeline-parallel recipes.
  • Hy3-preview. Adds a 295B MoE DeepEP fine-tuning and LoRA recipe, with model-specific expert-bias handling.
  • Hy-MT2-30B-A3B. Adds a supervised fine-tuning recipe and native model support.
  • MiniMax-M2.7. Adds pipeline-parallel full-SFT and LoRA recipes.
  • Falcon-H1. Adds 0.5B, 1.5B, and 7B full-SFT recipes, plus 7B and 34B PEFT recipes.
  • MiniCPM5-1B. Adds full-SFT and PEFT recipes.
  • Nemotron-3-Ultra-550B-A55B. Adds a 128-GPU throughput benchmark and a 128-GPU full-SFT recipe using a real router, THD sequence packing, and a repeated MTP head.
  • Qwen3.5/3.6 dense and MoE. Adds 27B dense and 35B-A3B VLM SFT/LoRA recipes, a 35B EP8/CP2 long-context recipe, MTP support, and a native dense Qwen3.5 backbone that keeps its SSM-gate parameters in fp32.

• VLM

  • Qwen3.6 VLM. Adds 27B dense and 35B-A3B SFT/LoRA recipes, including a 4K MedPix EP8/CP2 recipe for the MoE model.
  • Mistral Medium 3.5. Adds a 128B MedPix full-finetuning recipe and a matching LoRA recipe.
  • LLaVA-OneVision 1.5. Adds a 4B full-finetuning recipe and an 8B LoRA recipe.
  • VLM QLoRA. Enables the existing quantization: config block for VLM finetuning, including BitsAndBytes 4-bit NF4 base weights with LoRA adapters.
  • VLM knowledge distillation. Adds a Qwen3.5-VL 9B-to-4B KD recipe with a chunked KD loss, frozen vision/audio towers, and separate CE/KD metrics.
  • Step-3.7-Flash. Adds support for Stepfun AI’s MoE VLM model Step-3.7-Flash with SFT and LoRa recipes using EP + PP across 16 nodes.

• Omni and Multimodal

  • Bagel. Adds the Bagel model, dataset, pretraining recipe, finetuning recipe, EMA support, checkpoint adapter, and distributed initialization. The one-node examples use token budgets validated for parity runs.
  • Qwen2.5-Omni ASR. Adds AMI and multi-language SFT recipes for the 3B and 7B checkpoints.
  • Qwen3-Omni ASR. Adds AMI and multi-language audio SFT recipes.
  • Nemotron-Omni vision. Adds a dynamic-resolution image path aligned with vLLM's image preprocessing, preventing training/rollout vision-embedding divergence.
  • Nemotron-Omni parallelism. Adds activation checkpointing for the nested language model and an EP8/CP2 VLM recipe that prepares and shards multimodal embeddings across the context mesh.

• Diffusion

  • FLUX.2-dev. Adds a model adapter, dataset processor, full-finetuning recipe, LoRA recipe, and generation configuration for its 4D positional-ID and Mistral3 text-embedding architecture.
  • Qwen-Image. Adds model and processor support, full-finetuning, pretraining, generation, and LoRA recipes. The LoRA recipe targets attention and image/text MLP layers, and the data collator pads variable-length cached prompt embeddings.
  • Wan 2.2 T2V-A14B. Adds preprocessing, high-noise and low-noise stage finetuning, and generation that loads independently trained checkpoints into the matching denoisers.
  • FLUX, Hunyuan, and Wan 2.1. Adds LoRA recipes for each workflow, a multinode Wan 2.1 recipe, and tuned full-finetune/LoRA settings including compile and FSDP reduction-dtype controls.

• Gemma 4 Optimizations

  • TP/PP VLM support. Adds TP4/PP2 and TP4/PP4 recipes for Gemma 4 31B, including pipeline-stage handling for image position IDs.
  • LoRA coverage. Adds PEFT recipes for the 2B, 4B, 31B, and 26B-A4B MoE Gemma 4 VLM models.
  • Joint drafter. Adds a Gemma 4 joint-drafter model, MedPix recipe, and MTP inference benchmark.

• Retrieval and Embeddings

  • Late-interaction retrieval. Adds multi-vector/MaxSim training for ColBERT-style encoders, including distributed in-batch negatives.
  • Bi-encoder positives. Adds cycling over every positive passage instead of choosing only one positive document per query.
  • Embedding and reranker recipes. Adds a Ministral 3 3B bi-encoder recipe and model-coverage pages for bidirectional Ministral and Llama models.

• Agent SFT and Speculative Decoding

  • Agent-chat data adapter. Accepts ChatML messages and ShareGPT conversations, merges consecutive tool calls, pairs tool responses to call IDs, and supervises assistant content and tool-call arguments.
  • Final-turn SFT. Adds train_on_last_turn_only to mask every earlier assistant response and train only on the final assistant turn.
  • Reasoning traces. Preserves reasoning_content during data conversion and adds mask_reasoning_content to include the trace in the prompt while excluding it from loss.
  • Tool-call evaluation. Adds generation-based validation metrics for call presence, tool-name accuracy, JSON validity, argument precision/recall, and exact argument match.
  • EAGLE recipes. Adds EAGLE 1/2 recipes for Llama, Phi, Qwen3, and Qwen3-MoE; EAGLE 3 recipes for Llama, Phi, Qwen3, Qwen3-MoE, and GPT-OSS; and EAGLE 3.1 recipes for Llama.
  • Remote EAGLE target. Adds a remote target server/client path and an example that trains a draft model while target supervision runs on a separate GPU.
  • P-EAGLE sequence partitioning. Adds a sequence_partitions knob that splits a long-context draft step into loss-equivalent segments to reduce activation memory.
  • EAGLE packing and reasoning controls. Adds packed variable-length training with block-causal attention and a switch to keep, save, or mask target-model reasoning traces.
  • EAGLE performance tools. Adds an SGLang acceptance/speedup benchmark, offline target-output caching, and a fused Triton soft-target cross-entropy kernel that avoids allocating a full fp32 log-probability tensor.

• dLLM

  • DFlash. Adds a draft-training strategy that distills frozen target hidden states with a position-decay loss, with Nemotron Nano 30B and Qwen3-4B example recipes.
  • LLaDA2. Adds a discrete-diffusion SFT recipe alongside the existing LLaDA workflow.
  • Nemotron-Labs-Diffusion. Adds a hybrid diffusion-LLM SFT recipe.

• Training, Parallelism, and Performance

  • Selective FSDP2 activation checkpointing. Adds distributed.activation_checkpointing: selective, which saves attention and communication operations while alternating save/recompute for matrix multiplies and grouped MoE GEMMs. It supports dense models, MoE with expert parallelism, single-GPU runs, and torch.compile.
  • Memory-efficient LoRA. Adds use_memory_efficient_lora for LoRA training configurations.
  • Qwen3.5/3.6 VLM context parallelism. Prepares image/video embeddings and mRoPE positions before sequence sharding, passes the dense sequence index to linear attention, and uses a CP-aware validation denominator.
  • Nemotron v3 parallelism. Adds context and pipeline parallelism support for Nemotron v3 models and MTP-related train-loop support.
  • Cross-entropy kernels. Adds fused linear cross-entropy for custom models and a CUDA Triton implementation of EAGLE soft-target cross-entropy.
  • DeepSeek-V4 kernels. Adds TileLang implementations for sparse MLA, the lightning indexer, and MHC Sinkhorn; the CUDA environment now installs TileLang and TileKernels for the DeepSeek-V4 recipes.
  • Large-checkpoint loading. Uses memory-mapped local HF DCP reads so each rank maps only its requested tensor slice instead of copying full tensors into host RAM.

• Data, Recipes, and Operations

  • Object storage. Adds S3 and multi-storage-client sources to the Megatron pretraining dataset path.
  • Dataset blends. Adds data_dir_list entries in [num_samples, path] form for deterministic per-source sampling before concatenation, while retaining plain-path entries.
  • Blend files. Adds support for flat-list blend JSON files.
  • Lazy preprocessing. Adds lazy dataset preprocessing to defer work until it is needed by the data pipeline.
  • Retrieval runtime tuning. Exposes DDP options for bucket size, static graph, buffer broadcast, unused parameters, and gradient bucket views; exposes FSDP2 reshard_after_forward; and adds averaged retrieval-los...

Release Notes

• Highlights

  • Expanded VLM line-up: Gemma 4, Mistral 4, Qwen3.5 VL
  • Diffusion and discrete-diffusion LLM (new tracks)
  • NeMo Retriever – bi-encoder + cross-encoder / reranker
  • Knowledge Distillation scaled to TP > 1 and PP (Sepehr Sameni)
  • MoE infrastructure deepening – UCCL-EP, HybridEP, grouped_mm
  • SkyPilot launcher backend (Aditya Saxena, community)
  • End-to-end checkpoint + convergence robustness framework

• Model Support – newly supported families in r0.4.0

  • LLM
    • GLM 5 (#1372)
    • GLM 5.1 (#1720)
    • MiniMax M2.5 (#1280)
  • VLM / OMNI
    • Gemma 4 family – 2B, 4B, 31B, 26B-A4B MoE (#1658, #1660, #1731)
    • Mistral Small 4 (#1556)
    • Qwen3.5 VL dense – 4B, 9B (#1427)
    • Qwen3.5 VL MoE – 35B (#1373)
  • Diffusion
    • Flux T2I, Hunyuan T2V, Wan 2.1 T2V (see "Diffusion" section)
  • Discrete diffusion LLM
    • LLaDA (see "Discrete Diffusion LLM" section)

• Diffusion – new track in r0.4.0

  • HuggingFace Diffuser integration
  • r0.4.0 ships full pretrain / finetune / generate pipelines with LoRA support for diffusion models (T2V, T2I)
  • Wan integrated with multi-resolution DataLoader (#1475)
  • Inference utility for diffusion (#1491)
  • LoRA for diffusion (#1653, Linnan Wang)
  • Diffusion processor registry (#1379)
  • Models / recipes shipped
    • Flux T2I – pretrain, SFT, LoRA, generate
    • Hunyuan T2V – SFT, LoRA, generate
    • Wan 2.1 T2V – pretrain, SFT, LoRA, generate
  • Documentation guides for dataset preprocessing and finetuning.

• Discrete Diffusion LLM (dLLM) – new track in r0.4.0

  • Discrete diffusion LLM SFT support added (#1665)
  • LLaDA SFT recipe (#1672)
  • dLLM generation pipeline (#1692)

• NeMo Retriever (bi-encoder + cross-encoder)

  • Refactored cross-encoder / reranker training loop (new in r0.4.0) — (#1449).
  • Bi-encoder datasets can be loaded directly from the HuggingFace Hub (#1380)
  • Bi-encoder masking + consistent attn_implementation default (#1349)
  • Resolve retrieval dataset corpus paths relative to training file (#1367)
  • Docs: docs/guides/retrieval/finetune.md

• Knowledge Distillation — Sepehr Sameni

  • Enable TP > 1 in KD (#1297)
  • TP-aware KDLoss with distributed softmax + T² scaling (#1499)
  • Pipeline-parallelism support for KD (#1500)

• Parallelism / Performance / Train-loop

  • FSDP2
    • FSDP2 weight prefetching + async TP optimization (#1711)
  • Context Parallel
    • Qwen3.5 dense & MoE CP (#1710, #1560 — alexchiu / Zhaopeng Qiu)
    • Mamba CP for hybrid Nemotron v3 (#1441)
    • 3D mRoPE position_ids sharding under CP (#1482)
    • CP attention-mask hooks for dense / non-TE (#1470)
  • Pipeline Parallel
    • PP shape-inference optimization + pp_seq_len field in PipelineConfig (#1195, #1390)
    • Variable length for PP (#1689 – Zhiqi Li & Hemil Desai)
  • Activation checkpointing
    • Gradient_checkpointing overhead reduction i[n transformers 5.3 (#1621 — Yuki Huang)
  • MoE infrastructure
    • UCCL-EP alternative dispatcher (#1635 – Zhiqi Li & Hemil Desai)
    • HybridEP (#1333, #1666)
    • DeepEP-on-H100 RDMA fallback detection (#1275 — Piotr Żelasko)
    • torch._grouped_mm expert backend (#1228)
    • TE FusedAdam QuantizedTensor compatibility patch (#1417)
    • MoE LoRA rank scaling + torch_mm path (#1300, #1392)
    • Expert / diversity metrics (#1232, #1506), top-k utilization (#1418)
    • Packed sequences for MoE with EP+PP (#1685)
  • FlashOptim integration (#1492)
  • Scheduler-driven python GC (#1391)
  • fp32 RMSNorm backend + cast_model_to_dtype for improved stability (#1493)
  • Native Comet ML experiment tracking (#1411, Logan Vegna, community)
  • Added .generate() with KV-cache for Nemotron v3 (#1332, Piotr Żelasko)
  • Added output_hidden_states for NemotronHForCausalLM (#1386, Desh Raj)

• Launcher & CLI

  • SkyPilot backend (#1590 — Aditya Saxena, community contributor)
  • CLI app + launching refactor (#1406)
    • Shim scripts under examples/ will be deprecated post 26.04.
  • Launcher CLI flags no longer leak into recipe YAML overrides (#1766)
  • MFU logging in train recipes (#1413 — SwekeR, community)

• Checkpoint and convergence robustness

  • Checkpointing: End-to-end finetune → vLLM-deploy testing (#1606)
    • Models covered:
      • Gemma 3
      • Nemotron (Flash 1B, Super v3, Nano 9B, Nano v3)
      • Phi 4, Llama 3.2, Qwen 2.5
      • Qwen 3 MoE, GPT-OSS.
    • What this catches: prediction divergence, packaging gaps, vLLM loading issues.
  • Convergence harness (#1554, #1577, #1602)
    • Pipeline: Tulu-3 data prep → model verification → training → eval
    • Models covered:
      • GPT-OSS 20B (FlashAdamW + TE FusedAdam).
      • Moonlight 16B (3 configs incl. EP8+CP2).
      • Qwen3 4B (3 configs incl. CP1/CP2 variants).
      • Qwen3 MoE 30B (2 configs + experiments/).

• Datasets

  • Neat packing (greedy knapsack) for LLM and VLM (#1485 – Zhiqi Li)
  • Pretokenization support for VLM.(Zhiqi Li)
  • MultiImage dataset support for Qwen family (Zhiqi Li)
  • Qwen family video training support (Zhiqi Li)
  • LengthGroupedSampler (#1618 – Zhiqi Li)
  • Chat datasets THD/BSHD + CP, padding fixes (#1416).
  • reasoning_content + tool-calling support in ChatDataset (#1644, Zeel Desai, community).
  • Custom chat_template override for VLM finetuning (#1525, Bambuuai, community).
  • NEFTune noisy embeddings (#1686, stanley1208, community).
  • JSONL malformed-line skip (#1694, Somshubra Majumdar).

• Documentation

  • Per-model coverage pages (#1683).
  • Diffusion docs (#1495).
  • Gemma 4 tutorial (#1657).
  • Nemotron Parse fine-tuning notebook + assets (#1655, Krishna Kalyan).
  • Finetune-process + container-usage docs (#1484, Krishna Kalyan).
  • MLflow/Databricks docs (#1170, Andrei Onel).

• Contributions – we are grateful for all contributions 🙇

  • Khazzz1c
    • optimized resolve_yaml_env_vars from scanning runtime data in instantiate() (#1827)
    • additional contributions in r0.5.0.
  • Logan Vegna: added native Comet ML experiment tracking support (#1411).
  • Harsha Pasham: fixed error with aten::equal operator on meta tensors (#1769).
  • Aditya Saxena: added SkyPilot support. (#1590).
  • SwekeR-463:
    • Added MFU logging in train recipes (#1413).
    • Added embeddings utility functions for 15 models (#1288).
  • stanley1208
    • Implemented NEFTune noisy embeddings for fine-tuning (#1686).
    • Added best_metric_key field in CheckpointingConfig (#1641).
  • Zeel Desai
    • Added reasoning_content and tool-calling support to ChatDataset (#1644).
    • Additional contributions in the next release.
  • Bambuuai: enabled custom chat_template override for VLM fine-tuning (#1525).
  • Zakir Jiwani: Fixed instantiation issue in yaml parsing (issue #1496) (#1654).

• Known Issues

  • Minor memory regression in cohere_command_r_7b_hellaswag_fp8 and glm_4_9b_chat_hf_hellaswag_fp8
  • Qwen3_5_4b_neat_packing hangs during checkpoint saving
  • MegatronFSDP support postponed for 26.06
  • ~2% of checkpoint loading currently exercise a less-optimized path, which is being addressed in follow-up work.

Release Notes

• Hugging Face Transformers v5
  • Upgraded to Transformers v5 with a new device-mesh-only model initialization API
  • Drop-in API compatibility: NeMoAutoModelForCausalLM, NeMoAutoModelForImageTextToText, NeMoAutoModelForSequenceClassification, NeMoAutoTokenizer mirror the standard Transformers Auto* APIs
• Model Support
  • LLM
    • DeepSeek V3.2
    • Step 3.5 Flash
    • MiniMax M2
    • Nemotron-3-Nano v3 (30B-A3B)
    • Nemotron Flash 1B
    • GLM 4.7, GLM 4.7 Flash
    • Devstral-Small-2-24B
    • FunctionGemma (tool-calling)
    • Ministral3 (3B, 8B, 14B)
  • VLM & OMNI
    • Kimi-VL-A3B
    • Kimi K2.5 VL
    • Nemotron Parse v1.1
    • Qwen3 VL MoE (30B, 235B)
    • Ministral3 VLM (3B, 8B, 14B)
  • Embedding & Retrieval
    • NeMo Biencoder training pipeline with Llama-Embed-Nemotron-8B support
    • Hard negative mining for retrieval training
• PEFT
  • DoRA (Weight-Decomposed Low-Rank Adaptation)
  • LoRA for MoE models (DeepSeek MoE, Qwen MoE)
  • LoRA support for Biencoder
• Parallelism
  • Pipeline parallelism for VLMs
  • GroupedExpertsTE backend (prerequisite for MoE FP8)
  • TE RoPE fusion for custom MoE models
  • Norm fusion and RoPE cache for dense models
• Dataset support for
  • VLM multi-turn chat
  • Inline text dataset format for retrieval
  • Databricks DeltaLake streaming dataset
  • Parquet file support for Megatron dataset preprocessing
  • xLAM tool-calling dataset
  • Answer-only masking in ColumnMappedDataset
• Checkpointing & logging
  • Streaming safetensors writer for reduced peak memory during checkpoint saving
  • Explicit restore_from for checkpoint loading (replaces auto-loading behavior)
  • Checkpoint custom model code files alongside weights
  • Configurable remote logging frequency via step_scheduler
• Optimizers
  • Dion optimizer (Muon/orthogonal family)
• Performance
  • Faster FP8 dequant kernels for DeepSeek V3
  • Meta device initialization enabled by default for reduced peak memory during model setup
  • Combine projection refactor for dense models
  • SDPA as default attention backend when FlashAttention is unavailable
• Misc Infrastructure
  • Databricks integration (DeltaLake datasets, Unity Catalog checkpointing, DBFS consolidation)
  • Nsys profiling support with model layer name scoping
  • Environment variable dereferencing in YAML configs
  • Improved import time
• Resolved from 0.2.0
  • MoE perf regression with DeepSeek V3 (resolved via faster FP8 dequant kernels and GroupedExpertsTE backend)
  • PEFT (LoRA) support for MoE models (now available)
  • Validation for packed sequences with TE attention (fixed)
  • Validation support for pipeline parallelism (added)
• Known Issues
  • Qwen3-next unsupported on blackwell due to FLA lacking support.
  • TransformerEngine’s Fused Adam is not working with DTensor, resolved in the upcoming version.
  • LoRA with TE backend is not supported

Community Contributions

• We gratefully acknowledge the following contributions from the OSS community:
  • @onel (Andrei Onel) – Founder of @askmanu, Dublin
    • docs: Add documentation for the new ChatDataset class (#990)
    • docs: Added MLflow guide (#1045)
    • docs: Created guide for quantization aware training (#1088)
    • docs: Documentation update for release 0.2.0 (#1041)
    • docs: Update docs/guides/dataset-overview.md (#1145)
  • @ooooo-create – Community contributor (PaddlePaddle ecosystem)
    • fix: Add DeepEP fallback logic and tests (#1000)
    • fix: respect trust_remote_code when building AutoConfig (#1007)
  • @Sparlitu – Community contributor
    • fix: leave num_epochs unset if max_steps is specified (#1107)
  • @yuhezhang-ai (Yuhe Zhang) – Engineer at Polarr
    • feat: Support LoRA for custom MoEs (#1010)
    • fix: sequence classification metric and training bugs #780 (#841)
  • @therealnaveenkamal (Naveenraj Kamalakannan) – NYU graduate student
    • feat: Implement DoRA (#1150)
  • @dongs0104 (Dong Shin) – Samsung Research, Samsung Electronics
    • fix: resolving errors in the hf decorator function (#983)
  • @jbross-ibm-research (Juergen Bross) – IBM Research

• Fast Model Implementations
  • LLM
    • GPT-OSS 20B and 120B
    • Qwen3 next and Qwen3-235B
    • GLM-4.5-344BA32B, GLM-4.6, GLM-4.5-Air
  • VLM & OMNI
    • Qwen3-vl
    • Qwen2-5-vl
    • Qwen3-omni-30b-a3b
    • Intern-vl-4B (ootb)
• Parallelism
  • Improved support for CP and sequence packing with MoE models
  • Optimized TP plan for LoRA
• Dataset support for
  • Single-turn tool calling
  • Multi-turn tool calling
  • Streaming dataset
  • Chat dataset with OpenAI format
  • Improved support for truncation/padding
• Checkpointing & logging
  • Support for asynchronous checkpointing with DCP
  • Symbolic links (LATEST, LOWEST_VAL) pointing to the latest and lowest validation score checkpoints
  • MLFlow support
• Task support
  • QAT for SFT
  • Sequence classification
• Known issues
  • Minor perf regression with DSv3
  • Sequence parallel plan incorrect for Qwen3
  • Support for GPT-OSS 120B with DeepEP will be included in the next patch release
  • Validation is not functional for custom models with TE when using packed sequence and pipeline parallel size of 1
• Limitations
  • PEFT (LoRA) support for MoE models is scheduled for the 26.02 release
  • For non-MoE models, CP support requires the model leveraging the PyTorch SDPA API

NeMo-Automodel 25.11 Container

The 0.2.0 release is also included the NeMo Automodel 25.11 container on NGC at https://registry.ngc.nvidia.com/orgs/nvidia/containers/nemo-automodel.
Here are the major software components included in the container:

• Features:

  • Included support for limiting the number of samples with the ColumnMappedDataset

• Bug Fixes (step scheduler):

  • Switched to zero-based indexing
  • Epoch length accounts for accumulation steps

New Features

• Pretraining support for
  • Models under 40B with PyT FSDP2
  • Larger models by applying PyT PP
  • TP can also be used for models with a TP plan
  • Large MOE via custom implementations
• Knowledge distillation for LLMs (requires same tokenizer)
• FP8 with torchao (requires torch.compile)
• Parallelism
  • HSDP with FSDP2
  • Auto Pipelining Support
• Checkpointing
  • Pipeline support (load and save)
  • Parallel load with meta device
• Data
  • ColumnMapped Dataset for single-turn SFT
  • Pretrain Data: Megatron-Core and Nano-gpt compatible data
• Performance https://docs.nvidia.com/nemo/automodel/latest/performance-summary.html
  • Pretraining benchmark for Large MoE user-defined models
  • Fast DeepSeek v3 implementation with DeepEP

• Megatron FSDP support
• Packed sequence support
• Triton kernels for LoRA

Prerelease: NVIDIA NeMo-Automodel 0.1.0rc0 (2025-09-17)