Mixture-of-Experts (MoE)

The sleep framework proposes a novel method for LLMs to retain knowledge through two distinct phases: memory consolidation and dreaming. This approach optimizes the learning landscape by using structured compression and synthetic gradient diversification to prevent catastrophic forgetting.

This research proposes a new MoE architecture called Meta Expert that shares experts across layers to mitigate parameter redundancy. The approach demonstrates that models can maintain high performance with significantly fewer parameters and experts compared to traditional hierarchical designs.

DeepSeek's new V4.1 Flash model introduces a massive 763B parameter architecture that optimizes for low latency and high throughput. By using N-gram parameters as a conditional memory module, the model achieves significant intelligence gains without the typical memory overhead.

This technical paper introduces the T1 reinforcement learning method to solve stability issues in long-horizon agent tasks. It specifically focuses on mitigating token drift and routing instabilities in large-scale Mixture-of-Experts models.

FreeToken is a novel serving system that optimizes MoE model inference on consumer-grade hardware by adapting to varying bandwidth and memory availability. It addresses the challenges of expert movement and heterogeneous resource management in agentic workloads.

PagedWeight optimizes the serving of MoE LLMs by implementing a dynamic quantization strategy that adapts to runtime memory pressure. It balances hardware efficiency with model accuracy by monitoring expert routing statistics and prompt-specific sensitivities.

ELDR optimizes MoE model serving in disaggregated environments by routing requests with similar expert activation patterns to the same decode workers. This approach reduces memory bandwidth bottlenecks and significantly improves decoding latency.