MoE (Mixture-of-Experts)

A sparse architecture that activates only a subset of expert sub-networks via gating during inference to balance compute cost an