vLLM

AutoRound is an open-source quantization toolkit for LLMs and VLMs, focused on high-accuracy low-bit inference across CPU, XPU, CUDA, and multiple deployment backends.

Open-source C++ and CUDA LLM inference engine inspired by vLLM, with a teaching-focused course that walks through model serving, batching, KV cache, and attention kernels.

An open-source reinforcement learning training loop for LLMs, spanning small CPU-friendly experiments like CartPole to async RLVR runs on GPU clusters. The repo highlights a compact codebase, quickstart commands, task customization, and support for algorithms like REINFORCE, PPO, GRPO, and RLOO.

A logical KV cache profiler and leak detector for PagedAttention engines like vLLM and SGLang, with a dashboard, live engine hook, CI checks, and Perfetto trace export.

A detailed setup guide for building a two-node AMD Strix Halo cluster with Intel E810 RDMA networking for distributed vLLM inference.

An open-source CLI for inspecting live LLM inference processes on NVIDIA GPUs. It breaks down VRAM usage by component, shows runtime and model details, and projects memory savings from quantization strategies.

An out-of-tree vLLM plugin that compresses KV cache with STAR-KV to fit more concurrent requests and longer contexts on the same GPU memory budget.

A detailed vLLM blog post explaining speculative decoding on AMD GPUs, with mechanics, supported drafting methods, configuration guidance, tuning advice, and benchmark discussion.

Self-hosted, Kubernetes-native platform for distributed multi-model LLM inference. It includes an OpenAI-compatible API, installer flow, and infrastructure-as-code deployment with support for air-gapped environments.

Galahad is an AI memory layer that saves and restores model KV cache, supports document recall, and plugs into vLLM, SGLang, and llama.cpp. The page also shows agent features like replay, branching, rewind, and fault-finding.

An open-source course and reference implementation for a self-scaling OCR pipeline on Kubernetes, using Rust for ingestion, vLLM for inference, Redis and KEDA for event-driven scaling, and MCP integration for agent use.

Google Developers Blog post about integrating DFlash, a diffusion-style speculative decoding framework, into the vLLM TPU ecosystem to improve LLM serving speed on TPU v5p.

A vLLM blog post explaining Semantic Router as an open serving-layer runtime for bounded micro-agents, with routing patterns like Confidence, Ratings, ReMoM, Fusion, and Workflows behind a single model API.

A public slide deck about vLLM-compile, a project focused on bringing compiler optimizations to LLM inference and speeding up torch.compile for vLLM workflows.

A Hugging Face paper page describing a memory layer for LLMs that stores KV state to encrypted NVMe and reloads it later without recomputing, with results from 50M-token experiments on Gemma models.