KV Cache

AsyncLLM introduces an asynchronous programming model for LLMs that leverages Python's asyncio to enable concurrent agentic behaviors. By using composable CacheBlocks, the framework allows multiple coroutines to share and view memory states without the overhead of re-encoding tokens.

The AI landscape is undergoing a fundamental transition from training-centric development to an era dominated by inference demands. This shift is driven by the rise of reasoning models and autonomous agents that require massive memory bandwidth.

PagedWeight optimizes the serving of MoE LLMs by implementing a dynamic quantization strategy that adapts to runtime memory pressure. It balances hardware efficiency with model accuracy by monitoring expert routing statistics and prompt-specific sensitivities.

The researchers present Self Gradient Forcing (SGF) to solve the lack of gradient flow in historical KV caches during autoregressive video generation. This method enables much more stable and consistent long-form video extrapolation without the massive memory overhead of full backpropagation.

ELDR optimizes MoE model serving in disaggregated environments by routing requests with similar expert activation patterns to the same decode workers. This approach reduces memory bandwidth bottlenecks and significantly improves decoding latency.