Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine

NVIDIA Transformer Engine has added JAX support for Mixture of Experts (MoE) models, providing optimized kernels for expert and router computation. The implementation specifically targets dropless MoE architectures, which maintain all expert paths during training. This optimization benefits large-scale AI model training frameworks like DeepSeek, Qwen, and Mixtral that utilize MoE design patterns.