An MoE layer contains many expert feed-forward networks, and a router sends each token to only a few of them.
Total parameters are large, but only a fraction is active per token, so quality improves at a lower compute cost per token than a dense model of the same size.
Trade-offs: all experts must sit in memory, routing needs load balancing, and training and serving are more complex.
Loading…