KV cache

A technical primer that explains how LLM inference works, covering weights in GPU memory, prefill vs. decode, the KV cache, and batching.

An out-of-tree vLLM plugin that compresses KV cache with STAR-KV to fit more concurrent requests and longer contexts on the same GPU memory budget.

A research repo that replays real Claude Code and Mooncake traces to test prefix-cache eviction policies and compare them against LRU.

Reference implementation for using Linux Pressure Stall Information to trim an LLM KV cache under memory pressure. The repo includes requirements, basic usage commands, a simulator, a llama.cpp runner, and benchmark scripts with example results.

Galahad is an AI memory layer that saves and restores model KV cache, supports document recall, and plugs into vLLM, SGLang, and llama.cpp. The page also shows agent features like replay, branching, rewind, and fault-finding.

A GitHub repository for an engine-agnostic KV cache fragment system for on-device LLM inference, with Android and Go components, adapters for llama.cpp/MLC-LLM/ONNX Runtime, and benchmark and monitoring code.

arXiv paper on a cache-merging method for multi-agent latent reasoning, framing KV-cache composition as a convergent replicated state with deterministic merging.

A Hugging Face paper page describing a memory layer for LLMs that stores KV state to encrypted NVMe and reloads it later without recomputing, with results from 50M-token experiments on Gemma models.