Video Generation · 3D/4D Reconstruction · AI Infra

I'm an M.Sc. student in Electrical Engineering and Information Technology at the Technical University of Munich and a Research Intern at Agile Robots SE. I hold an M.Sc. in Computer Science from Tongji University (2025).

My research focuses on video generation and 3D/4D reconstruction, from camera-controlled video diffusion and avatar animation to sparse-view reconstruction. My open-source work in AI infrastructure and CUDA/Triton GPU kernels complements this research through distributed execution, numerical correctness, and memory-efficient implementation.

• Agile Robots SE — Cross-Embodiment Video Generation
  Research Intern · Apr 2026–Present

  Developing cross-embodiment video generation to translate egocentric human demonstrations into robot-domain videos for policy learning in dexterous manipulation.

• TUM Visual Computing — Camera-Controlled Video Generation
  Master's Thesis · Feb 2026–Present

  Developing camera-controlled video diffusion to synthesize human motion and human–scene interactions across changing viewpoints.

• TUM Visual Computing — 3D-Aware Avatar Animation
  Guided Research · Apr 2025–Mar 2026

  Developed ViDS by conditioning video diffusion on 3D face tracking, enabling identity-preserving, long-form portrait animation from a single image.

• TUM Computer Vision Group — Sparse-View 3D Reconstruction
  Guided Research · Feb 2024–Aug 2025 · In collaboration with Oxford VGG

  Co-developed CSG-Fusion through matching-based pruning, mask-based feature aggregation, cross-view supervision, and experimental evaluation for compact, consistent sparse-view 3D reconstruction.

• ImFusion GmbH / TUM CAMP — Online Long-Term Video Tracking
  Research Intern · Aug 2024–Apr 2025

  Implemented online inference and EMA-flow initialization for LiteTracker, supporting low-latency, frame-by-frame tissue tracking in endoscopic video streams.

Selected engineering contributions span generative model execution, GPU kernels, and differentiable rendering. Each repository is an independent project.

• xDiT

  Technical details and selected PRs

  • Request state and collectives. Request-local state is reset while communication groups are reused. The dedicated-VAE path then creates those groups consistently across DiT ranks, preventing leaks and mismatched collective order.

    Request lifecycle #836 (Merged) → VAE-rank collectives #842 (Merged)

  • Cache validity. Shape guards prevent incompatible cache reuse; scheduler request hooks additionally reset step counts and residuals for same-shape requests. Regression and pipeline tests exercise consecutive generation calls.

    Shape compatibility #813 (Merged) → Request boundaries #864 (Merged)

  • Selected PRs.

    Mixed-length prompt masks #856 (Open)

  • Cache layout and execution ownership. Draft changes align PipeFusion warmup and patch-mode KV layouts, exclude Ideogram padding, and pad/unpad uneven HunyuanVideo shards. A separate runner fix assigns prompt splitting, output gathering, and parallel VAE ownership to one layer each, preventing double processing and missing outputs. These are independent execution contracts, with draft paths still under review.

    Patch-major KV cache #899 (Draft) · Ideogram attention bounds #900 (Draft) · HunyuanVideo shard boundaries #901 (Draft) · Runner and wrapper ownership #902 (Merged)

• SGLang · Native diffusion serving and cache lifecycle

  Merged request-cache resets and FLUX guidance fixes improve diffusion inference correctness; proposed Ovis-Image serving extends model integration and attention consistency.

  Technical details and selected PRs

  • Native Ovis-Image integration. A native pipeline connects checkpoint mapping, text conditioning, CFG, TP/SP, offload, and image serving. The PR reports bitwise agreement with pinned Diffusers for tested single-GPU exact-tier runs on B200; parallel runs are checked against explicit BF16 noise controls. Unsupported features fail with explicit errors.

    Ovis-Image model and serving #43152 (Open)

  • Request entry to expert state. Resetting all DiT caches when a request starts covers experts that never execute denoising step zero, preventing stale TeaCache/Spectrum state across requests.

    Per-request DiT reset #42437 (Merged)

  • FLUX true CFG. Uses true_cfg_scale for the native FLUX branch-combination weight, preserving existing behavior when it is unset. Unit tests and real FLUX.1-dev comparisons on B200 verify the correction; numerical differences from Diffusers remain.

    FLUX true CFG weight #44260 (Merged)

  • Sharded SDPA consistency. Eligible exact-tier torch-SDPA calls pad query rows so TP/Ulysses head shards select the unsharded attention kernel. B200 operator and two-GPU Ulysses tests report bitwise agreement; whole-model differences remain. The proposal retains the existing lossless-tier and unaffected backend behavior.

    Exact-tier head shards #43242 (Open)

• vLLM-Omni · Error propagation in diffusion serving

  Typed validation errors retain their original API status through the single-GPU diffusion executor.

  Technical details and selected PRs

  • Typed worker errors. Single-GPU execution retains client-error type and status instead of flattening exceptions into generic server failures, so API responses reflect the original validation error.

    Executor error metadata #8474 (Merged)

• DiffSynth-Studio · Reliable generation pipelines and memory-efficient execution

  Merged fixes improve VAE memory use, attention precision, conditioning alignment, and pipeline model residency.

  Technical details and selected PRs

  • Memory-efficient VAE execution. Merged Qwen-Image VAE tiling blends overlapping spatial tiles and handles image boundaries. With BF16 Qwen-Image weights, 2048 × 2048 images, 30 denoising steps, tile size/stride 128/64, and one B200, the PR reports incremental VAE decode memory falling from 8.26 to 2.11 GiB and encode memory from 7.51 to 1.88 GiB versus non-tiled execution. Untiled outputs remain bit-identical; tiled outputs differ slightly, with reported PSNR of 54.9 dB for text-to-image and 50.2 dB for image-to-image.

    Qwen-Image VAE tiling #1720 (Merged)

  • Attention precision and video conditioning. FLUX.2 casts Q/K to the value dtype after KV-cache concatenation, restoring BF16 FlashAttention execution under autocast. Real-weight FLUX.2-klein-base-4B LoRA training completes 50 finite-loss steps on a B200 with torch 2.12.0 and flash-attn 2.8.3.post1. An independent Wan VACE fix trims control videos and masks to num_frames; reported 480 × 832, 49-frame Wan2.1-VACE-1.3B outputs at 30 steps match pre-trimmed inputs bitwise. These are separate precision and conditioning contracts.

    FLUX.2 attention dtype #1718 (Merged) · Wan VACE frame bounds #1721 (Merged)

  • Model residency and dependency resolution. JoyAI-Image uses pipeline attribute names for VRAM onloading and split-training classification, and aligns edit-unit output declarations. Real-weight 1024 × 1024, 30-step B200 editing tests run with CUDA onloading and match CPU-onload outputs pixel-for-pixel. A separate uv packaging fix scopes the PyTorch CPU index, declares conflicting extras, and corrects the PyWavelets dependency. Packaging validation covers lock and dry-run dependency resolution, without claiming an Ascend-device execution test.

    JoyAI model onloading #1725 (Merged) · uv dependency resolution #1723 (Merged)

• Diffusers · Branch-aware video cache integration

  Proposed Wan cache isolation keeps conditional and unconditional guidance branches separate.

  Technical details and selected PRs

  • Conditional and unconditional contexts. Conditional and unconditional calls receive separate cache contexts, preserving classifier-free guidance branch isolation and enabling cached execution in the Wan pipeline.

    Wan V2V cache contexts #14948 (Open)

• FastVideo · Hardware-aware video inference and training

  Merged Triton layout fusion removes GPU passes; proposed GB10 inference and NVFP4 training extend hardware coverage.

  Technical details and selected PRs

  • SM100 and GB10 inference. A fused Triton kernel removes separate scatter and transpose passes. The PR reports 1.155× end-to-end speedup for Wan2.1-VSA-T2V-14B on one B200 with 50 steps, with bitwise-identical frames in the tested configurations. A proposed GB10/SM121 extension reuses the same fused layout for eligible BF16, tile-64, D128 inference. DGX Spark tests report exact paired outputs for the tested FastWan2.1 1.3B configurations, with memory-safety checks and full-model validation; the reported timing benefit varies by prompt.

    Fused VSA layout #1900 (Merged) → GB10 fused-layout support #1948 (Open)

  • GB10 / SM121 QAT. Padding-aware NVFP4 scales and SM121 split-path routing underpin forward-consistent dV. The proposed backward reconstructs forward quantization statistics; its training-gradient contract remains under evaluation.

    Quantization and routing prerequisites (#1933 (Open) + #1934 (Open)) → Forward-consistent dV #1938 (Open)

• RL-Kernel

  Technical details and selected PRs

  • Selected PRs.

    AdaLN projection #485 (Open) · Fused norm and modulation #487 (Open) · Sequence-parallel reductions #494 (Open) · Full-model drift controls #571 (Open) · Checkpoint mapping and rollout adapter #575 (Open)

  • Qwen3-Next normalization. The stack restores CUDA dispatch, specifies zero-centred normalization, adds gated forward/backward kernels, then defines GDN decode-state goldens. The final stage supplies PyTorch recurrent and convolution state references. Reference and provider checks expose casting and state differences before model integration.

    CUDA dispatch #466 (Merged) → Zero-centred RMSNorm #467 (Open) → Gated RMSNorm #468 (Open) → Decode-state contracts #469 (Open)

  • Qwen3-Next integration and reproducibility. The proposed stack connects MoE routing, TP4 model execution, rollout/recompute checks and fixed-order distributed logprobs. Recorded B200 reruns verify 97/97 batch-invariance cells and invariant digests at 17 TP4 points across 13 NCCL settings. Selected-token logprobs and LSE match TP1 on the same logits; entropy remains TP-degree dependent. An EP4 follow-up verifies 16/16 rollout/recompute cases and 97/97 batch-invariance cells. EP4 retains its own numerical profile, and its training and checkpoint-export paths remain disabled.

    MoE route/combine #503 (Open) → TP4 model mixers #507 (Open) → Rollout/recompute chain #543 (Open) → Batch-invariance validation #544 (Open) → Distributed logprob parity #549 (Open) → Expert-parallel rollout/replay #557 (Open)

  • Memory-efficient logp backward. A fused CUDA backward writes the returned gradient directly, with no extra temporary workspace; older builds retain a bounded chunked fallback. For BF16 logits with 32,768 rows and vocabulary 151,936 on one B200, the PR reports forward/backward peak memory falling from 46.37 to 9.27 GiB and median latency from 57.1 to 27.2 ms versus the previous VJP. The peak still includes the returned gradient.

    Fused logp backward #495 (Open)

  • Recursive deterministic autograd. An independent deterministic LogP path composes reusable VJP/JVP primitives and torch.func rules while retaining the first-order fast path. Fresh single-B200 tests cover derivative orders 1–8 in four precisions, with additional FP64 reverse-mode checks through order 10; hardware skips remain separate from passes. External performance for the expanded contract is unmeasured, and the memory figures above apply only to the generic backward proposal.

    Recursive LogP autograd #547 (Open)

• FlashAttention · SM100 attention kernels and validation

  Merged SM100 forward cache isolation prevents NaN contamination from unused V-cache entries; proposed work addresses backward correctness, scheduling, and reproducible validation.

  Technical details and selected PRs

  • dV memory transfers. Deriving V tensor-memory transfers independently of Q/K fixes the SM100 dV epilogue when forward succeeds but backward silently uses the wrong transfer width.

    Asymmetric-head dV #2975 (Open)

  • Cache-safe forward. Sequence-bounded nonpaged V loads zero-fill unused cache rows before P@V, preventing 0 × NaN contamination in dense and packed forward. The PR reports B200 correctness and memory-check coverage; valid V NaNs and paged-KV behavior retain their existing contracts.

    Unused V-cache isolation #2983 (Merged)

  • Scheduling and reproducible validation. Adapts an existing CLC L2-swizzle implementation while preserving STATIC 2CTA compatibility. A separate dense/varlen benchmark records CUDA Graph timings, source revisions, compiler identity, and kernel hashes. The report credits upstream work for the observed register-spill gains.

    CLC and STATIC scheduling #2976 (Open) · Dense/varlen validation tool #2982 (Open)

• gsplat · Robust differentiable rendering

  Proposed rasterization and loss fixes preserve valid outputs and gradients across degenerate inputs.

  Technical details and selected PRs

  • Rendering contracts. Empty scenes retain background outputs and gradient shapes; depth-mode backward handles strided gradients and median-depth channels; degenerate masked selections return a differentiable zero without leaking excluded NaN/Inf values. Targeted regressions cover packed/unpacked rendering, geometry gradients, and zero-loss backward.

    Empty-scene inputs #1079 (Open) · Depth gradients #1068 (Open) · Masked losses #1082 (Open)

  • Clamped-alpha geometry gradients. The 3DGUT backward proposal gates only the alpha-dependent path at the clamp, preserving hit-distance and normal terms for geometry gradients. PyTorch-reference comparisons and a synthetic depth-only optimization check support the correction; remaining scale growth follows the reference and reflects the underconstrained loss.

    Clamped-alpha distance and normals #1083 (Open)

• PyTorch3D · Camera visibility and bounded box overlap

  Proposed visibility and bounded box-clipping fixes prevent invalid feature sampling and CUDA buffer overflow.

  Technical details and selected PRs

  • Visibility contract. World-to-view depth rejects points behind a camera, preventing invalid projections from contributing sampled features and gradients.

    Camera visibility #2056 (Open)

  • Bounded box clipping. A bounded CUDA workspace first prevents triangle-buffer overflow. Polygon clipping then bounds each clipped triangle to nine vertices, removes that workspace, and aligns CPU/CUDA treatment of coincident faces. On one B200, the PR reports 100 × 100 box pairs dropping from 1.315 to 0.675 ms versus valid upstream controls, using the median of 31 CUDA-event samples; the 22-test box-IoU module passes.

    CUDA capacity bounds #2059 (Open) → Bounded polygon clipping #2060 (Open)

• NVIDIA Warp · Numerical stability and sparse GPU products

  Draft FP16 SVD and proposed zero-adjoint fixes address forward and backward NaNs; proposed sparse extensions add mixed-precision arithmetic and optional CUDA row packing.

  Technical details and selected PRs

  • SVD3 forward precision. Input scaling and FP32 decomposition produce half-precision factors while preserving public signatures and existing adjoints. Reported CPU/B200 tests cover zero, rank-deficient, repeated-singular-value, and scaled inputs. The forward-only contribution remains draft pending maintainer guidance.

    FP16 SVD3 forward #2063 (Draft)

  • Zero-adjoint SVD gradients. A separate proposal bypasses reciprocal singular-value gaps when both U/V adjoints are zero, preserving singular-value gradients and preventing unused SVD diagnostics from contaminating other gradients. CPU and B200 regressions cover repeated singular values and small nonzero seeds. Maintainer guidance remains pending; the general nonzero U/V adjoint formula is unchanged, and downstream impact is unmeasured.

    Zero-adjoint gradient contract #2064 (Open)

  • Mixed-precision sparse products. Sparse products keep matrix storage in FP32 while multiplying FP64 vectors, accumulating and scaling in FP64 without allocating a converted matrix. A dependent, optional CUDA row-packing path supports non-transposed products while preserving default dispatch. CPU and B200 correctness checks cover the proposed paths; performance depends on sparse structure, and no end-to-end solver speedup is claimed.

    Mixed-precision sparse arithmetic #2072 (Open) → Optional CUDA row packing #2073 (Open)

_{Selected public contributions · PR status verified 2026-10-26.}