Skip to content

feat: GPT-OSS 20B and Moonlight 16B convergence results - #1577

Merged
hemildesai merged 11 commits into
mainfrom
hemild/tulu3-convergence-models
Apr 2, 2026
Merged

feat: GPT-OSS 20B and Moonlight 16B convergence results#1577
hemildesai merged 11 commits into
mainfrom
hemild/tulu3-convergence-models

Conversation

@hemildesai

@hemildesai hemildesai commented Mar 19, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • Add GPT-OSS 20B convergence pipeline with custom {% generation %} chat template, training configs, W&B plots, and full results
  • Update Moonlight 16B README (remove parameter table, keep results)
  • Fix parallelizer.py to support num_local_experts attribute (used by GPT-OSS / Mixtral-style configs)

GPT-OSS 20B Results

Model prompt_strict inst_strict
Base 0.303 0.440
FlashAdamW 32-bit 0.484 0.622
TE FusedAdam FP32+BF16 0.468 0.603

Key findings:

  • {% generation %} tags wrapping full assistant output (channel header + content + stop token) improve IFEval by ~4pts over no-tag training
  • Abrupt ending reduced 39.4% to 12.2%
  • activation_checkpointing: false works well for GPT-OSS 20B (~58 GiB peak)
  • quantization_config must be removed from saved config.json before vLLM eval (pretrained model uses mxfp4 but SFT saves bf16)

Moonlight 16B Results

Model prompt_strict inst_strict
Pretrained 0.148 0.278
FlashAdamW 32-bit, fp32_upcast 0.412 0.559
TE FusedAdam FP32+BF16 0.381 0.534

Key findings:

  • SFT improves instruction following: prompt_strict 0.148 to 0.412 (+179%)
  • FlashAdamW 32-bit with fp32_upcast is the best configuration
  • Death loop rate (~20-22%) remains consistent -- needs further investigation
  • TikToken tokenizer requires slow tokenizer fallback patch for generation tag support

Test plan

  • GPT-OSS FlashAdamW training converges (loss 1.21 to 0.50, 1000 steps)
  • GPT-OSS TE FusedAdam training converges (loss 1.21 to 0.50, 1000 steps)
  • Moonlight FlashAdamW and TE FusedAdam training converges
  • IFEval eval with recommended sampling parameters
  • Inference quality analysis (death loop, abrupt ending, missing EOS, empty)
  • W&B plots generated and included for both models
  • Generation tag loss mask verified
  • Pre-commit hooks pass

Generated with Claude Code

@copy-pr-bot

copy-pr-bot Bot commented Mar 19, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@hemildesai
hemildesai force-pushed the hemild/tulu3-convergence-models branch 4 times, most recently from 14b8f49 to dd7cc0c Compare March 19, 2026 01:00
Base automatically changed from hemild/tulu3-convergence to main March 27, 2026 01:19
@hemildesai
hemildesai requested review from a team and pthombre as code owners March 27, 2026 01:19
@hemildesai

hemildesai commented Mar 30, 2026

Copy link
Copy Markdown
Contributor Author

/claude review

claude[bot]
claude Bot reviewed Mar 30, 2026
View reviewed changes
Comment thread nemo_automodel/components/datasets/llm/chat_dataset.py Outdated
Comment on lines +117 to +124
sl = None
if split is not None:
match = _SPLIT_SLICE_RE.match(split)
if match:
base_split = match.group(1)
start = int(match.group(2)) if match.group(2) else None
end = int(match.group(3)) if match.group(3) else None
sl = slice(start, end)

@claude claude Bot Mar 30, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: The split-parsing logic (regex match → base_split + sl) is now duplicated across the HF path (lines 81-89) and this Parquet path. Consider extracting a small helper to avoid the duplication. Not blocking, but worth noting since a bug fix to one would need to be applied to both.

@hemildesai

hemildesai commented Mar 30, 2026

Copy link
Copy Markdown
Contributor Author

/claude review

@hemildesai

hemildesai commented Mar 30, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 50b9bee

claude[bot]
claude Bot previously approved these changes Mar 30, 2026
View reviewed changes

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

hemildesai and others added 2 commits March 30, 2026 15:23
@hemildesai @claude
feat: add Moonlight 16B and GPT-OSS 20B convergence pipelines
856b76a 
Moonlight 16B (A3B):
- Custom chat_template.jinja with {% generation %} tags
- FlashAdamW and TE FusedAdam configs (EP=8, FSDP)
- TikToken slow tokenizer fallback patch
- Training curves and MoE metrics plots
- Results: prompt_strict 0.148 -> 0.412 (+179%)

GPT-OSS 20B:
- Custom chat_template.jinja with {% generation %} tags wrapping
  full assistant output (channel header + content + stop token)
- FlashAdamW and TE FusedAdam configs (EP=8, FSDP, AC=false)
- Training curves and MoE metrics plots
- Results: prompt_strict 0.303 -> 0.484 (+60%)

Fix parallelizer.py to search for num_local_experts attribute
(GPT-OSS uses Mixtral-style config) with unit test.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: hemildesai <hemild@nvidia.com>
@hemildesai @claude
refactor: extract _parse_split_slice helper to deduplicate split parsing
0b0522e 
Address review comment: the split-parsing regex logic was duplicated
between the HF and Parquet code paths in _load_openai_messages.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Hemil Desai <hemild@nvidia.com>
@hemildesai
hemildesai force-pushed the hemild/tulu3-convergence-models branch from 50b9bee to 0b0522e Compare March 30, 2026 22:23
@hemildesai

hemildesai commented Mar 30, 2026

Copy link
Copy Markdown
Contributor Author

/claude review

claude[bot]
claude Bot reviewed Apr 1, 2026
View reviewed changes
Comment thread examples/convergence/tulu3/models/moonlight-16b/README.md
claude[bot]
claude Bot reviewed Apr 1, 2026
View reviewed changes
Comment thread nemo_automodel/_transformers/tokenization/nemo_auto_tokenizer.py Outdated
@hemildesai @claude
refactor: rename _apply_chat_template_with_slow_fallback to _apply_ch…
de0e6b6 
…at_template_safe

Replace fast/slow tokenizer terminology with native HF vs TikToken
in docstrings and comments. Update test class and helper names to match.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: hemildesai <hemild@nvidia.com>
@hemildesai

hemildesai commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

/claude review

@hemildesai

hemildesai commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test de0e6b6

claude[bot]
claude Bot reviewed Apr 1, 2026
View reviewed changes
Comment thread examples/convergence/tulu3/models/moonlight-16b/README.md
@hemildesai @claude
feat: convert TikToken tokenizers to fast backend at load time
8189108 
Instead of relying on heuristic char-to-token text matching for
{% generation %} mask computation, convert TikToken-based tokenizers
to Rust-backed fast tokenizers in from_pretrained using TikTokenConverter.
This enables native char_to_token() support. The _apply_chat_template_safe
fallback is kept as a safety net. Also addresses claude bot review comments
(stale patch reference in README, magic number documentation).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: hemildesai <hemild@nvidia.com>
@hemildesai

hemildesai commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

/claude review

@hemildesai

hemildesai commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 8189108

claude[bot]
claude Bot previously approved these changes Apr 1, 2026
View reviewed changes

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@hemildesai @claude
refactor: remove _apply_chat_template_safe heuristic fallback
b2003c2 
The TikToken-to-native conversion at load time makes char_to_token()
work natively, so the heuristic text-matching fallback is no longer
needed. Also rename _try_convert_tiktoken_to_fast to
_try_convert_tiktoken_to_native.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: hemildesai <hemild@nvidia.com>
@hemildesai

hemildesai commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

/claude review

@hemildesai

hemildesai commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b2003c2

claude[bot]
claude Bot reviewed Apr 1, 2026
View reviewed changes
Comment thread examples/convergence/tulu3/models/moonlight-16b/README.md Outdated
claude[bot]
claude Bot reviewed Apr 1, 2026
View reviewed changes
Comment thread nemo_automodel/_transformers/tokenization/nemo_auto_tokenizer.py Outdated
@hemildesai @claude
fix: remove remaining autocast blocks and update stale README reference
46550a8 
Remove autocast blocks from moonlight-16b and gpt-oss-20b configs that
were missed in the earlier cleanup. Update moonlight README to reference
_try_convert_tiktoken_to_native instead of the removed
_apply_chat_template_safe.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: hemildesai <hemild@nvidia.com>
@hemildesai

hemildesai commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

/claude review

@hemildesai

hemildesai commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 46550a8

@akoumpa

akoumpa commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

/ok to test 5d7d65f

@hemildesai @claude
docs: remove stale _apply_chat_template_safe reference in docstring
eb7c5ad 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: hemildesai <hemild@nvidia.com>
@hemildesai

hemildesai commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

/claude review

@hemildesai

hemildesai commented Apr 1, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test eb7c5ad

claude[bot]
claude Bot approved these changes Apr 1, 2026
View reviewed changes

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

This branch was previously deployed

2 inactive deployments
nemo-ci — eb7c5adc Deployed Apr 2, 2026 by copy-pr-bot[bot] via L2_Datasets #4948
test — eb7c5adc Deployed Apr 1, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #4948
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adil-a adil-a adil-a left review comments
@akoumpa akoumpa akoumpa approved these changes
@claude claude[bot] claude[bot] approved these changes
@HuiyingLi HuiyingLi Awaiting requested review from HuiyingLi HuiyingLi is a code owner
@snowmanwwg snowmanwwg Awaiting requested review from snowmanwwg snowmanwwg is a code owner
@ZhiyuLi-Nvidia ZhiyuLi-Nvidia Awaiting requested review from ZhiyuLi-Nvidia
@pthombre pthombre Awaiting requested review from pthombre

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hemildesai @akoumpa @adil-a