GPU

This article details how AI is unevenly accelerating scientific progress, with major impacts in cybersecurity and minor effects in mathematics. It also highlights two new frameworks, SPADE for synthetic environment generation and Hawkeye for automated GPU kernel optimization.

OasisKV addresses the memory wall in LLM inference by implementing a sparse prefetching mechanism for the KV cache. By leveraging speculative decoding to predict future token importance, it moves less critical KV data to cheaper memory tiers without stalling the decode process.

PagedWeight optimizes the serving of MoE LLMs by implementing a dynamic quantization strategy that adapts to runtime memory pressure. It balances hardware efficiency with model accuracy by monitoring expert routing statistics and prompt-specific sensitivities.