KV-cache

PagedWeight optimizes the serving of MoE LLMs by implementing a dynamic quantization strategy that adapts to runtime memory pressure. It balances hardware efficiency with model accuracy by monitoring expert routing statistics and prompt-specific sensitivities.

The researchers present Self Gradient Forcing (SGF) to solve the lack of gradient flow in historical KV caches during autoregressive video generation. This method enables much more stable and consistent long-form video extrapolation without the massive memory overhead of full backpropagation.

PagedWeight optimizes the serving of MoE LLMs by implementing a dynamic quantization strategy that adapts to runtime memory pressure. It balances hardware efficiency with model accuracy by monitoring expert routing statistics and prompt-specific sensitivities.

The researchers present Self Gradient Forcing (SGF) to solve the lack of gradient flow in historical KV caches during autoregressive video generation. This method enables much more stable and consistent long-form video extrapolation without the massive memory overhead of full backpropagation.

ELDR optimizes MoE model serving in disaggregated environments by routing requests with similar expert activation patterns to the same decode workers. This approach reduces memory bandwidth bottlenecks and significantly improves decoding latency.

ELDR optimizes MoE model serving in disaggregated environments by routing requests with similar expert activation patterns to the same decode workers. This approach reduces memory bandwidth bottlenecks and significantly improves decoding latency.