What is MoE?

MoE is a model design with many small expert modules, where only a few are chosen for each input. The model is large overall but uses only part of it per answer, which keeps costs down.