Home About

Tags

MachineLearning (26) LinearAlgebra (16) GameDesign (12) ComputerScience (11) SoftwareEngineering (11) LargeLanguageModels (9) AudioEngineering (7) DiscreteMath (6) AutomatedTesting (6) Roblox (5) Zettelkasten (5) AgenticReasoning (4) More

Notes by Lex Toumbourou

Sparse Mixture of Experts Model

Oct 15, 2024 permanent

Sparse Mixture of Experts Model are models with a router component that sends request to a subset of layers. Mixtral 8x7B is an example.