KV cache

A technical primer that explains how LLM inference works, covering weights in GPU memory, prefill vs. decode, the KV cache, and batching.

An out-of-tree vLLM plugin that compresses KV cache with STAR-KV to fit more concurrent requests and longer contexts on the same GPU memory budget.

Reference implementation for using Linux Pressure Stall Information to trim an LLM KV cache under memory pressure. The repo includes requirements, basic usage commands, a simulator, a llama.cpp runner, and benchmark scripts with example results.

A GitHub repository for an engine-agnostic KV cache fragment system for on-device LLM inference, with Android and Go components, adapters for llama.cpp/MLC-LLM/ONNX Runtime, and benchmark and monitoring code.

arXiv paper on a cache-merging method for multi-agent latent reasoning, framing KV-cache composition as a convergent replicated state with deterministic merging.