[fusheng-ji]

RFC #420 H3 stack 5/5 (base test-h3, merge in this order): #586 -> #587 -> #588 -> foundation -> projection rows -> WS2 parallel -> token refiner -> full transformer. This PR merges after its parent; until then GitHub also shows the parent's commits. This PR's own diff.

Evidence location: the measurement outputs of this PR (JSON, figures) are not part of its diff. They are on the fork's pr_assets branch under pending/h3-full-transformer/ (links in this description are fixed to that commit). Paths such as reports/experiments/... below name where a runner writes its output.

Acceptance evidence (maintainer): Hopper H100 (sm90) on pr_assets @ d38c2a87. Blackwell (B200) below is supplementary (includes inherited #570 B200 pytest gap and B200-only dual-trajectory amplification).

What

• Full 33B forward/backward with strict boundary rows (audio_proj_in, audio_proj_out, context_embedder, token_refiner) bound to RL-Kernel and the provider boundaries
  (proj_in, proj_out, time_embedder, norm_out) kept explicit; full_model_execution(..., strict_boundaries=True) records which boundary used which binding.
  h3_full_transformer.py runs the provider twice (and the SDPA-efficient / cuBLASLt variants) and the candidate twice, with GPU-side checksum fingerprints.
• Dual trajectory (h3_dual_trajectory.py): the same Euler sampler run single (rollout-style, one sample) and batched (train-style) and recorded as sealed trajectories;
  --strict-only and --diag-video-projections select the arms. This is the rollout-vs-train control.

Prior art & reuse decision

diffusers is the provider and the only reference for the full model; vLLM-Omni's rollout path and the training path are what the dual trajectory stands in for. Nothing offers a
per-step rollout-vs-train drift report, so the harness is new.

Results (H100 / Hopper)

Full 33B step time (forward + backward, gradient checkpointing, one H100) — strict graph binds h3_full_attention (#553) in CandidateOps when _H3_C is built; provider is diffusers/cuBLAS. Median of 3 repeats; torch 2.13.0+cu130, MiniMax-H3@42ed227, runner tools/validation/models/h3_full_speed.py, measured at head 4f7c5d11 (#575 stack, includes #571).

tokens	provider total	strict total	total ratio	provider fwd	strict fwd	fwd ratio	strict peak GiB
12	0.312 s	1.3 s	4.03x	0.104 s	0.42 s	4.0x	62.9
925	0.438 s	5.0 s	11.33x	0.119 s	1.31 s	11.0x	63.0
4253	1.709 s	37.4 s	21.88x	0.438 s	9.10 s	20.8x	63.5
8349	3.625 s	126.4 s	34.88x	0.910 s	28.70 s	31.6x	65.0

Strict uses the bounded-workspace #553 attention kernel (not the interim quadratic DeterministicAttentionOp); 8349 tokens fit in ~65 GiB. This is still far above provider time (launch-bound strict path plus interim projection/GEMM rows in the stack); it does not meet the “R/R step time not worse than P/P” acceptance bar.

Measured on NVIDIA H100 (93 GiB, sm90), driver 580, CUDA 13.0, torch 2.13.0+cu130, pinned MiniMax-H3@42ed227 weights, at head 5ae9f5eb. Strict vs provider on the 33B forward/backward (tiny request): candidate bitwise repeatable True, provider repeatable True;
video output relative difference 2.07e-03, audio 1.77e-02, input-gradient relative differences 0.23, 0.23, 0.33 (BF16 provider vs FP32-accumulated candidate; gradients of a 33B network amplify rounding differences).

Rollout (single) vs train (batched) path, 4 Euler steps x 4 comparisons, three seeds:

Seed	Provider: differing comparisons (first max abs)	RL-Kernel strict	Strict + video projections bound (diagnostic)
11	16/16 (first 0.47)	8/16 (first 3.8e-06)	0/16
22	16/16 (first 0.25)	8/16 (first 5.2e-06)	0/16
33	16/16 (first 0.54)	8/16 (first 3.8e-06)	0/16

Numbers: reports/experiments/h3-full-transformer-h100/.

Hopper pytest summary: tests.json (142 collected, 142 passed @ eb7840bb, H100 sm90).

Dual trajectory on Hopper (strict, no diagnostic binding): 8/16 comparisons differ on every seed; per-step max abs on video velocity stays ~1e-5–1e-6 through step 3 (e.g. seed 33 step 3 ≈ 5.7e-06, not the B200 spike). Provider single-vs-batched drift is large from step 0 and grows each step (seed 33 step 3 provider video-velocity max 2.578 (dual_trajectory_step_max.json)). With --diag-video-projections, strict is 0/16 bitwise equal on seeds 11/22/33 (*_diag.json). Summary: dual_trajectory_diag_summary.json.

RFC H-C2 (Hopper): full strict single-vs-batched bitwise match is not claimed until walker-ai video proj_in/proj_out rows are bound in production strict mode; the diagnostic arm shows layout drift is attributable to those rows.
The provider's two paths differ at every step (H100: max abs grows from ~0.545 (step 0) to ~2.578 (step 3) on seed 33 on seed 33; the table's "first" column is step-0 only). Strict first differs by 4e-6 to 5e-6, which comes from the video proj_in / proj_out rows (another contributor's, not bound here);
with them bound only as a diagnostic the two paths are bitwise equal at every step on all three seeds. The wall-time panel includes warm-up; steady-state step time is not claimed here.

Results (B200 / Blackwell) — supplementary, not Hopper acceptance

Measured on NVIDIA B200 (179 GiB, sm100), driver 580.126, CUDA 13.0, torch 2.13.0+cu130, pinned MiniMax-H3@42ed227 weights, clean tree at eb7840bb.

Tests: 141 passed, 1 failed (142 collected); the one failure is the token-refiner accuracy case inherited from #570 (test_nodes_accuracy_repeat_batch_and_isolated[1-257], swiglu node, 0.01211 against a bound of 0.01116; see #570). The bound was not changed. tests.json.

Strict vs provider on the 33B forward/backward (tiny request; the result does not depend on the seed, the three runs are identical): candidate bitwise repeatable True, provider repeatable True; video output relative difference 2.29e-03, audio 2.13e-02, input-gradient relative differences 0.20 (video hidden states), 0.19 (audio), 0.33 (encoder); 638 parameter gradients, relative difference median 0.20, maximum 0.55. Peak allocated 68.2 GiB. (full_transformer_seed11.json, seeds 22 and 33 alongside.)

Rollout (single) vs train (batched) path, 4 Euler steps x 4 comparisons. Cells are differing comparisons out of 16, with the largest absolute difference at steps 0 / 1 / 2 / 3 in brackets (0 = bitwise equal):

Seed	Provider	RL-Kernel strict	Strict + video projections bound (diagnostic)
11	16/16 (3.3e-06 / 3.3e-06 / 4.3e-06 / 6.7e-06)	8/16 (3.8e-06 / 4.9e-06 / 4.1e-06 / 7.6e-06)	0/16
22	16/16 (3.8e-06 / 3.8e-01 / 6.9e-01 / 9.6e-01)	8/16 (3.1e-06 / 4.5e-06 / 3.9e-06 / 5.7e-06)	0/16
33	16/16 (3.1e-06 / 3.8e-06 / 8.6e-06 / 6.3e-06)	10/16 (4.8e-06 / 5.2e-06 / 5.4e-06 / 6.2e-01)	0/16

Reading the table:

• The provider's two paths differ at every step. Its drift is about 1e-6 in two seeds and jumps to 0.38 at step 1 in seed 22 (0.96 at step 3).
• Strict is not bitwise equal between the layouts on B200 without the diagnostic binding (8, 8 and 10 of 16 comparisons differ). It stays at about 1e-6 for the first steps and, in seed 33, jumps to 0.62 at step 3. The first difference is at step 0 in all three seeds and comes from the video proj_in / proj_out rows, which are not bound here.
• With those two rows bound as a diagnostic, strict is bitwise equal at every step on all three seeds, as on H100.
• The wall-time panel includes warm-up and the provider run absorbs most of it; steady-state step time is not claimed.

Root cause (B200-focused) — observed vs step-3 probe

Observed (artifact-backed):

• Unbound video proj_in / proj_out: strict layout drift ~1e-6 from step 0; --diag-video-projections → 0/16 on H100 and B200.
• B200 strict seed 33 step 3 video-velocity max 0.6215; H100 same case 5.72e-06 (dual_trajectory_step_max.json).

Euler step-3 probe (seed 33, handoff script on pr_assets; not a #571 code commit):

• H100 seed33_report.json: arms A/B/C step-3 video velocity max all ~5.7e-06; delta-scale K=1000 on step-2 latent fork → ~0.53 (mechanism reference on sm90 only).

• B200 seed33_report.json @ eb7840bb, driver 580.126, torch 2.13.0+cu130; reused strict single/batched safetensors from the B200 [WS1][CUDA][MiniMax-H3] ws1_full_h3_transformer and ws1_dual_trajectory: 33B forward/backward and rollout-vs-train control (RFC #420 stack 5/5) #571 dual-trajectory run:

  • Step-2 latent fork (D): video 1.9e-06, audio 0.
  • Arm A (natural step-3 entry): video velocity max 0.6215, audio 0.305 (reproduces historical dual-trajectory spike).
  • Arms B/C (shared step-2 latent; single vs batched layout): video 4.3e-06 / 5.4e-06, audio 0 — when entry latents match, layouts agree; not “layout alone” at step 3.

• H100 step-3 probe retest (2026-10-23): seed33_report.json, layerwise_report.json; pytest 142/142 @ eb7840bb (tests.json).

• Summary JSON: dual_trajectory_diag_summary.json.

Interpretation (not Euler accumulation, and not a device sensitivity difference):

• The model casts the FP32 latent to bfloat16 before proj_in. At the step-3 input of seed 33 on B200 the two layouts' video latents differ by 1.9e-06 in FP32, and after the cast 1 of 384 elements differs (index [0,1,8], 0.40625 against 0.408203125, one bf16 step, 1.95e-03); the audio latent and every other entry latent that was checked are identical after the cast (README.md, Correction).
• That one element reproduces the whole spike: with only it changed (same layout, arm P) the step-3 velocity gap is 0.6215 (video) and 0.305 (audio), on every block identical to the natural pair (arm A). With the same input in both layouts (arms B, C) the gap is 4.3e-06 / 5.4e-06. The audio gap comes from the video side through the joint attention (the audio latent is identical).
• Layer by layer (layerwise_report.json): the relative difference of the block output is 7.1e-04 after block 0 and 0.22 after block 49 (about 300 times), growing by 1.1 to 1.3 times in most blocks; there is no single sensitive layer. The largest velocity gap, 0.62, is about 4% of the reference maximum (14.6).
• On H100 the same seed has a 2.6e-06 FP32 fork but 0 of 384 elements differ after the cast at step-3 entry (337/384 differ in FP32 only; steps 0–2 also 0 bf16-differing cells — step_entry_bf16_audit_full.json, seed33_step3_entry_bf16_audit.json). H100 layerwise retest (2026-10-23, mcml): arm A/B 5.7e-06 (layout-only); arms E/P 0 on every block because there is no bf16 cell to perturb (layerwise_report.json). The two devices are therefore not shown to differ in sensitivity; B200 seed 33 straddles a bf16 rounding boundary. H100 K=1000 (~2.6e-03 latent delta, 0.53 velocity) remains the same order as one bf16 step (1.95e-03, 0.62).
• The probe script's conclusion_b200_style=supports_latent_accumulation is a B≪A threshold only, not a mechanism label. Dividing the velocity gap by the FP32 latent difference is not a gain of the forward (the cast removes that difference on H100 and turns it into one bf16 step on B200); the effective one on B200 is about 0.62 / 1.95e-03 ≈ 320.
• Limitations: one seed; one run per arm; B200 one-element bf16 perturbation (arm P) with layerwise depth profile; H100 layerwise confirms E/P trivially zero (no bf16 cell at step-3 entry) — not an artificial one-element forward on H100; H100 K-sweep not run on B200.

Status

Hopper (merge gate): full 33B strict/provider harness, repeatability, and dual-trajectory evidence on H100 at pr_assets@d38c2a87; strict layout drift ~1e-6 on unbound video projections; diagnostic binding yields 0/16 on H100.

B200 (supplementary): inherits #570's single pytest fail; strict dual trajectory seed 33 step 3 ~0.62, caused by one bf16 step in one input element of the step-3 forward (1 of 384 video elements differs after the bf16 cast; with identical inputs the layouts agree to ~5e-06), then spread over the 50 blocks; H100 seed 33 has no differing element after the cast — see Interpretation and dual_trajectory_diag_summary.json. Not a Hopper merge blocker unless maintainers require sm100 parity.

Tests

python -m pytest tests/models/minimax_h3/test_h3_full_model.py tests/models/minimax_h3/test_h3_token_refiner.py tests/models/minimax_h3/test_h3_operator_trace.py \
    tests/models/minimax_h3/test_h3_arch_fingerprint.py tests/models/minimax_h3/test_h3_projection_rows.py tests/models/minimax_h3/test_h3_one_block.py -q   # 136 passed
python tools/validation/models/h3_dual_trajectory.py --checkpoint <hf_full> --seed 11 --output <dir>

B200: same files, 141 passed, 1 failed (the failure is the #570 swiglu case).

Scope / Known limitations

Evidence priority: RFC #420 CUDA review should treat H100 / Hopper results as acceptance; B200 is additional sm100 signal only.

Strict mode is fail-closed: no TF32, no fast-math, no split-K; FP32 accumulation with one final rounding; unsupported shapes, dtypes and devices raise instead of falling back. Not part of this PR: video projections (#562), O / gate_up / down GEMMs (#554, #535), full attention (#553), ROCm, SGLang adapter, pipeline parallel. The full-model strict path still binds interim attention and O / gate_up / down stand-ins, so its step time is several times the provider's; speed is not claimed.

[coderabbitai]

Important

Review skipped

Too many files!

This PR contains 155 files, which is 55 over the limit of 100.

To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch.

Upgrade to a paid plan to raise the limit.

This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.

Check out review usage here.

Run configuration

• Source mode: Auto
• Repository: RL-Align/RL-Kernel
• Branch: test-h3
• File: .coderabbit.yaml
• Commit: d22d4b4
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Advanced
• Run ID: 281dc368-469e-497e-8993-bb834d03ee10

Commits

Reviewing files that changed from the base of the PR and between d22d4b4 and f069b29.

Files selected for processing: 155

• .github/workflows/ci.yml
• benchmarks/models/benchmark_h3_conditioning.py
• build_tools/extensions.py
• csrc/bindings/ops.cpp
• csrc/cuda/h3/adaln_row_gather.cu
• csrc/cuda/h3/det_linear.cu
• csrc/cuda/h3/gate_residual.cu
• csrc/cuda/h3/rmsnorm_modulate.cu
• csrc/cuda/h3/timestep_sinusoid.cu
• docs/.nav.yml
• docs/operators/README.md
• docs/operators/h3-adaln-gate-residual.md
• docs/operators/h3-adaln-projection.md
• docs/operators/h3-adaln-row-gather.md
• docs/operators/h3-arch-fingerprint.md
• docs/operators/h3-final-adaln-out.md
• docs/operators/h3-full-transformer.md
• docs/operators/h3-one-block.md
• docs/operators/h3-operator-trace.md
• docs/operators/h3-projection-rows.md
• docs/operators/h3-rmsnorm.md
• docs/operators/h3-sp-norm-adaln.md
• docs/operators/h3-timestep-mlp.md
• docs/operators/h3-timestep-sinusoid.md
• docs/operators/h3-token-refiner.md
• docs/operators/h3-tp-adaln-3mod.md
• docs/operators/h3-trajectory-artifact.md
• examples/models/minimax_h3/README.md
• examples/models/minimax_h3/candidate.sbatch
• examples/models/minimax_h3/diagnose-block.sbatch
• examples/models/minimax_h3/full-model-ensemble-rule.json
• examples/models/minimax_h3/full-model-thresholds.json
• examples/models/minimax_h3/generation-compare.sbatch
• examples/models/minimax_h3/generation-contract.json
• examples/models/minimax_h3/generation-inputs.json
• examples/models/minimax_h3/generation-prompt.txt
• examples/models/minimax_h3/generation-resume-check.sbatch
• examples/models/minimax_h3/generation.sbatch
• examples/models/minimax_h3/reference-manifest.json
• examples/models/minimax_h3/reference.sbatch
• rl_engine/_C.pyi
• rl_engine/backends/cuda/model_specific/__init__.py
• rl_engine/backends/cuda/model_specific/minimax_h3/__init__.py
• rl_engine/backends/cuda/model_specific/minimax_h3/adaln_modulation.py
• rl_engine/backends/cuda/model_specific/minimax_h3/adaln_projection.py
• rl_engine/backends/cuda/model_specific/minimax_h3/adaln_row_gather.py
• rl_engine/backends/cuda/model_specific/minimax_h3/det_linear.py
• rl_engine/backends/cuda/model_specific/minimax_h3/final_adaln_out.py
• rl_engine/backends/cuda/model_specific/minimax_h3/gate_residual.py
• rl_engine/backends/cuda/model_specific/minimax_h3/projection.py
• rl_engine/backends/cuda/model_specific/minimax_h3/rmsnorm.py
• rl_engine/backends/cuda/model_specific/minimax_h3/sp_norm_adaln.py
• rl_engine/backends/cuda/model_specific/minimax_h3/timestep_mlp.py
• rl_engine/backends/cuda/model_specific/minimax_h3/timestep_sinusoid.py
• rl_engine/backends/cuda/model_specific/minimax_h3/tp_adaln_projection.py
• rl_engine/backends/cuda/model_specific/minimax_h3/ws2_comm.py
• rl_engine/backends/shared/triton/gemm/h3_projection.py
• rl_engine/models/minimax_h3/arch_fingerprint.json
• rl_engine/models/minimax_h3/fingerprint.py
• rl_engine/models/minimax_h3/operator_trace.py
• rl_engine/models/minimax_h3/pinned_checkpoint.json
• rl_engine/models/minimax_h3/trajectory.py
• rl_engine/reference/minimax_h3/__init__.py
• rl_engine/reference/minimax_h3/adaln_projection.py
• rl_engine/reference/minimax_h3/adaln_row_gather.py
• rl_engine/reference/minimax_h3/block.py
• rl_engine/reference/minimax_h3/final_adaln_out.py
• rl_engine/reference/minimax_h3/fixed_order.py
• rl_engine/reference/minimax_h3/gate_residual.py
• rl_engine/reference/minimax_h3/projection.py
• rl_engine/reference/minimax_h3/rmsnorm.py
• rl_engine/reference/minimax_h3/timestep_mlp.py
• rl_engine/reference/minimax_h3/timestep_sinusoid.py
• rl_engine/runtime/registry.py
• rl_engine/validation/models/h3_block.py
• rl_engine/validation/models/h3_cases.py
• rl_engine/validation/models/h3_chain.py
• rl_engine/validation/models/h3_full_manifest.json
• rl_engine/validation/models/h3_full_model.py
• rl_engine/validation/models/h3_generation_resume.py
• rl_engine/validation/models/h3_manifest.json
• rl_engine/validation/models/h3_provider.py
• rl_engine/validation/models/h3_refiner.py
• rl_engine/validation/models/h3_report.py
• rl_engine/validation/models/h3_weights.py
• rl_engine/validation/models/h3_ws2.py
• rl_engine/validation/operators/operator_inputs.py
• rl_engine/validation/operators/operator_specs.py
• tests/build/test_build_platform_collectives.py
• tests/models/minimax_h3/conftest.py
• tests/models/minimax_h3/test_h3_adaln_gate_residual.py
• tests/models/minimax_h3/test_h3_adaln_modulation.py
• tests/models/minimax_h3/test_h3_adaln_projection.py
• tests/models/minimax_h3/test_h3_adaln_row_gather.py
• tests/models/minimax_h3/test_h3_arch_fingerprint.py
• tests/models/minimax_h3/test_h3_benchmark_timing.py
• tests/models/minimax_h3/test_h3_chain_replay_cli.py
• tests/models/minimax_h3/test_h3_cli.py
• tests/models/minimax_h3/test_h3_conditioning_e2e.py
• tests/models/minimax_h3/test_h3_det_linear.py
• tests/models/minimax_h3/test_h3_final_adaln_out.py
• tests/models/minimax_h3/test_h3_full_model.py
• tests/models/minimax_h3/test_h3_generation_resume.py
• tests/models/minimax_h3/test_h3_native_validation.py
• tests/models/minimax_h3/test_h3_one_block.py
• tests/models/minimax_h3/test_h3_operator_trace.py
• tests/models/minimax_h3/test_h3_projection_rows.py
• tests/models/minimax_h3/test_h3_report.py
• tests/models/minimax_h3/test_h3_rmsnorm.py
• tests/models/minimax_h3/test_h3_sp_norm_adaln.py
• tests/models/minimax_h3/test_h3_timestep_mlp.py
• tests/models/minimax_h3/test_h3_timestep_sinusoid.py
• tests/models/minimax_h3/test_h3_token_refiner.py
• tests/models/minimax_h3/test_h3_tp_adaln.py
• tests/models/minimax_h3/test_h3_trajectory_artifact.py
• tests/models/minimax_h3/test_h3_weights.py
• tests/models/minimax_h3/test_h3_ws2_processes.py
• tests/models/minimax_h3/test_prepare_h3_weights.py
• tests/runtime/test_dispatch.py
• tests/validation/operators/test_ws1_gtest_gpu.py
• tools/migration/README.md
• tools/migration/layout.json
• tools/validation/models/h3_bitwise_reuse.py
• tools/validation/models/h3_block_diagnostic.py
• tools/validation/models/h3_block_replay.py
• tools/validation/models/h3_block_ulp_sweep.py
• tools/validation/models/h3_candidate_prepare.py
• tools/validation/models/h3_chain_replay.py
• tools/validation/models/h3_dual_trajectory.py
• tools/validation/models/h3_evidence.py
• tools/validation/models/h3_evidence_integrity.py
• tools/validation/models/h3_fingerprint.py
• tools/validation/models/h3_first_step_diagnostic.py
• tools/validation/models/h3_full_charts.py
• tools/validation/models/h3_full_compare.py
• tools/validation/models/h3_full_ensemble_rule.py
• tools/validation/models/h3_full_generation.py
• tools/validation/models/h3_full_model.py
• tools/validation/models/h3_full_provider_spread.py
• tools/validation/models/h3_full_transformer.py
• tools/validation/models/h3_generation_adopt.py
• tools/validation/models/h3_generation_compare.py
• tools/validation/models/h3_operator_trace.py
• tools/validation/models/h3_prior_art.py
• tools/validation/models/h3_projection_bench.py
• tools/validation/models/h3_projection_charts.py
• tools/validation/models/h3_refiner_report.py
• tools/validation/models/h3_trajectory.py
• tools/validation/models/h3_ws2_evidence.py
• tools/validation/models/plot_h3_bitwise_reuse.py
• tools/validation/models/plot_h3_block.py
• tools/validation/models/plot_h3_evidence.py
• tools/validation/models/plot_h3_prior_art.py
• tools/weights/prepare_h3_weights.py
• tools/weights/snapshot_h3_checkpoint.py

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}