vLLM

This research highlights the critical failure of using black-box LLMs as evaluators due to inherent non-determinism in shared computing environments. It advocates for a rigorous, preregistered auditing framework to ensure measurement stability and reproducibility in AI evaluation.

This technical report examines the efficacy of using Random Attention for KV cache eviction to optimize LLM inference throughput. The study shows that this method significantly boosts tokens per second in high-concurrency scenarios without sacrificing model accuracy.

OasisKV addresses the memory wall in LLM inference by implementing a sparse prefetching mechanism for the KV cache. By leveraging speculative decoding to predict future token importance, it moves less critical KV data to cheaper memory tiers without stalling the decode process.

TokTier is a novel stateful tokenization service that optimizes LLM inference for agentic workloads by avoiding redundant full-text re-tokenization. It employs incremental repair for session updates and GPU-accelerated processing for new contexts, significantly reducing time to first token.

ELDR optimizes MoE model serving in disaggregated environments by routing requests with similar expert activation patterns to the same decode workers. This approach reduces memory bandwidth bottlenecks and significantly improves decoding latency.

This report details the rising trend of opportunistic scanning targeting AI-related services and infrastructure. It highlights specific threats to unauthenticated Ollama instances and the use of coordinated sweeps to harvest AI API keys from configuration files.