Sparse Mixture of Experts Model

Sparse Mixture of Experts Model are models with a router component that sends request to a subset of layers. Mixtral 8x7B is an example.

Oct 15, 2024 permanent

Sparse Mixture of Experts Model are models with a router component that sends request to a subset of layers. Mixtral 8x7B is an example.