llm-serving

An open-source handbook for production LLM serving and inference at scale, covering GPU fundamentals, KV cache, batching, quantization, speculative decoding, and engines like vLLM, SGLang, and TensorRT-LLM.

Flux is a Rust-based LLM inference planner and runtime that measures local hardware, chooses a model placement plan, and serves models behind an OpenAI-compatible API.

A research repo that replays real Claude Code and Mooncake traces to test prefix-cache eviction policies and compare them against LRU.

A detailed vLLM blog post explaining speculative decoding on AMD GPUs, with mechanics, supported drafting methods, configuration guidance, tuning advice, and benchmark discussion.

Galahad is an AI memory layer that saves and restores model KV cache, supports document recall, and plugs into vLLM, SGLang, and llama.cpp. The page also shows agent features like replay, branching, rewind, and fault-finding.