Wentao Guo @WentaoGuo7

🚀 Kernel Design Agents to Optimize Kimi Delta Attention (upto 2.96x) When the KDA (Kernel Design Agents) project was initiated, its name happened to coincide with another KDA (Kimi Delta Attention). More than once, people asked us , "Can we use KDA to write a version of KDA?" Today, right at a moment with a full moon good for shooting, we are thrilled to share the latest results of KDA(gent)-v0.6: using KDA(gent) to optimize KDA(ttn)! KDA-v0.6 achieves this through * Humanize2 Flow & LLM Collaboration Optimization: Introducing multi-round iterative evolution with Flame Chase, GPT-5.6-Sol, and Fable-5, paired with an automated Workspace cleanup mechanism to significantly broaden the operator search space. * Multilingual Domain-Specific Skill Stack Support: Fully supporting CuTe-DSL, CUDA C++, as well as recent Agent-Native CAKE IR and TIRx. Combined with customized diagnostic tools (such as CuTe's dedicated IKET pipeline analyzer and TIRx's CPU-side numerical simulation and static checks), achieving deep performance bottleneck diagnostics. * Self-Evolving Kernel Wiki: Building a dynamically updated knowledge base that automatically cleans up invalid/erroneous implementations, corrects classification tags, and streamlines search indexing, significantly improving the Agent's knowledge retrieval quality and code generation accuracy. KDAgent ultimately optimized the KDAttn kernel to 2.96x of FlashKDA, and we face a lot of unexpected hacks during the optimization. More details shared in our blogs nvlabs.github.io/kda/blog/2026-… Release results of KDA for KDA: github.com/NVlabs/kda/tre… Community & Production Deployment: All kernels are open-sourced for community verification! For deploying NVIDIA Agentic CUDA acceleration in production environments, we recommend using the production-level version rigorously verified by FlashInfer. By the Team: Dongyun Zou, Yixin Dong, Hongyi Jin, Junxian Guo, Yahui Cui, Avery Huang, Zihao Ye, Junru Shao, Changye Li, Zijian Zhang, Sihao Liu, Song Bian, Ligeng Zhu.

⚡ Attention kernels are well-optimized on GPU. But what about TPU? 🎯 We release Flywheel Attention: TPU-native softmax attention (MHA / GQA) + Gated DeltaNet kernels. Flywheel is 1.56× faster than the best open-source MHA/GQA kernel and 3.61× faster than the best open-source GDN kernel we could find. 👇

We pretrained a 2.3B MoE (360M active) Hybrid Mamba-2 that lands within a few points of Llama-3.2-3B using <1% of its pretraining FLOPs. No dedicated cluster. The run hopped between H100s, A100s, V100s (yes, V100s) and TPU v5p/v6e on a single codebase. Meet Rigel 🧵

Open-source video generation is now faster than playback without compromising quality. Introducing Video Delta Net (VDN): hybrid attention for live text-to-video with near-lossless quality. VDN accelerates Minimax-H3 by 75 - 90 x, generating 14 seconds of 768p video in 11 seconds on 8× NVIDIA B200 GPUs. Checkpoints + training/inference code + Technical Blog ⬇️ (1/6)

@percyliang I am so glad to see that!

@rosstaylor90 Great catch, thanks! This failed case only surfaces at large scale, so it slipped past our unit tests and previous experiments. We'll add coverage for it. Cool to see sonic-moe kernel is getting run at this scale🙏

🎉 Thrilled to share that our paper, “Thought-Level Beam Search for Reasoning,” has been accepted to COLM 2026! Different from Self-Consistency and Pruning-only approaches, Gambit actively reallocates test-time compute while generation is in flight — pruning low-scoring reasoning traces and branching from the strongest prefixes. See thread 🧵

keyboard_arrow_left Previous keyboard_arrow_right Next

Robot policies can move but can't think. LLMs can think but can't move. So we connected them. Real robot: 16.7% → 97.3% Sim (LIBERO-PRO): 12.8% → 53.3%

keyboard_arrow_left Previous keyboard_arrow_right Next

This was a large cross-domain effort, made possible by collaborators across systems, language models, robotics, AI for Science, trustworthy learning, and more. Huge thanks to everyone who helped build, validate, and test MLS-Bench: Bohan Lyu, Yucheng Yang, Siqiao Huang @KnightNemo_, Jiaru Zhang, Qixin Xu, Xinghan Li @XinghanLi66, Xinyang Han, Yicheng Zhang, Huaqing Zhang @zhqwqwq, Runhan Huang @RunhanH, Kaicheng Yang, Zitao Chen, Wentao Guo @WentaoGuo7, Junlin Yang @junlin45300, Xinyue Ai @KeelyAi04, Wenhao Chai @wenhaocha1, Yadi Cao @YadiCao, Ziran Yang @__zrrr__, Kun Wang @KunWang0129, Dapeng Jiang, Huan-ang Gao, Shange Tang @sangertang1999, Chengshuai Shi @chengshuai_shi, Simon S. Du @SimonShaoleiDu, Max Simchowitz @max_simchowitz, Jiantao Jiao @JiantaoJ, Dawn Song @dawnsongtweets, and Chi Jin @chijinML. (8/9)

Why do we store the SSM state at all? More and more models are hybrids (Nemotron-3, Qwen3.5), so SSM decode speed matters. We only write it back every step so the next step can read it. ReplaySSM caches the recent inputs instead and rebuilds the state on the fly. Same outputs, half the memory traffic → ~2x on spec decode at large batch sizes, which barely even helped SSMs before → up to 1.43x standard decode on large hybrids (up to Nemotron-Ultra-550B) Work with @tri_dao Blog + Code👇

@LijieyYang Nice!

Excited to share that LessIsMore has been accepted to ICML 2026! 🚀 LessIsMore is a training-free sparse attention for efficient long-horizon reasoning. By enforcing cross-head unified token selection, it brings up to 1.6x E2E speedup while preserving reasoning accuracy under practical workloads. Huge thanks to my amazing co-authors and mentors @Jackfram2, @JiaZhihao, Ravi! Paper: arxiv.org/abs/2508.07101 Code: github.com/DerrickYLJ/Les… #ICML2026 #LLM #EfficientAI

@IlysMoutawwakil I ran the benchmark on DGX B300 GPUs (Blackwell Ultra, technically SM103).

🚀SonicMoE🚀now runs at peak throughput on NVIDIA Blackwell GPUs 😃 54% & 35% higher fwd/bwd TFLOPS than the DeepGEMM baseline and 21% higher fwd TFLOPS than the triton official example. SonicMoE still maintains its minimum activation memory footprint: the same as a dense model with equal activated parameters and independent of expert granularity. We wrote a blogpost on how we leveraged Blackwell features and the software abstraction on QuACK: Work with @MayankMish98, @XinleC295, @istoica05, @tri_dao

@GoonGarrett Could you try it sometime? I guess it is the metadata computation that causes the hang before, but if it still appears I will take a look.

Again, great thanks to NVIDIA CUTLASS team @nvidia, PLI @PrincetonPLI and AI Lab, SkyLab @BerkeleySky and Together AI @togethercompute

[7/N] A detailed blogpost of our approach describing how we use Blackwell's TMEM double-buffering, 2CTA MMA, CLC tile scheduling, and gather fusion to hide IO costs behind MMA compute. We also walk through QuACK's software abstraction, and a few ablation studies on Blackwell GPUs. Blogpost: dao-lab.ai/blog/2026/soni…

[6/N] All SonicMoE Grouped GEMM kernels are built on QuACK. Each kernel overrides a single function. The most complex kernel (dH backward) only adds ~200 LoC for SonicMoE on top of QuACK, and the same code runs on Hopper and Blackwell GPUs.

[5/N] On Hopper, we leverage Ping-Pong warpgroup scheduling to overlap the heavy epilogue IO with the tiled GEMM computation. On Blackwell, the hardware does it differently but in the same spirit: we have a dedicated on-chip tensor memory (TMEM) split into two accumulator stages. The MMA warp fills one stage while epilogue warps handle the other stage, then they swap. The heavy epilogue IO is overlapped with the tiled GEMM again, which we elaborate on in the blogpost.

keyboard_arrow_left Previous keyboard_arrow_right Next

[4/N] SonicMoE is designed for fine-grained MoEs. It achieves greater relative speedup over existing MoE baselines when we increase the expert granularity on B300 GPUs.

[3/N] On B300 GPUs, SonicMoE achieves 54% & 35% higher fwd/bwd TFLOPS than the DeepGEMM baseline and 21% higher fwd TFLOPS than the triton official example across 6 open-source MoE configs (7B to 685B). SonicMoE often doubles the achieved TFLOPS over ScatterMoE and MoMoE.