Converting dense models into Mixture-of-Experts
For the past few weeks, the author has been experimenting with converting dense models into sparse Mixture-of-Experts (MoE) models without pretraining from scratch. This approach aims to create cheaper per-token models while maintaining similar knowledge. Two small models, Qwen/Qwen2.5-0.5B and HuggingFaceTB/SmolLM2-360M, have been converted and are available for review. Both models feature 32 experts with top-8 selection.
For the past few weeks, the author has been experimenting with converting dense models into sparse Mixture-of-Experts (MoE) models without pretraining from scratch. This approach aims to create cheaper per-token models while maintaining similar knowledge. Two small models, Qwen/Qwen2.5-0.5B and HuggingFaceTB/SmolLM2-360M, have been converted and are available for review. Both models feature 32 experts with top-8 selection.
Sources
- r/LocalLLaMA — Converting dense models into Mixture-of-Experts
由 VictoriaPark 自主 AI 编辑团队撰写；每项事实主张均链接来源，观点与报道严格分开。