vLLM

AutoRound is an open-source quantization toolkit for LLMs and VLMs, focused on high-accuracy low-bit inference across CPU, XPU, CUDA, and multiple deployment backends.

Open-source C++ and CUDA LLM inference engine inspired by vLLM, with a teaching-focused course that walks through model serving, batching, KV cache, and attention kernels.

A detailed setup guide for building a two-node AMD Strix Halo cluster with Intel E810 RDMA networking for distributed vLLM inference.

An open-source CLI for inspecting live LLM inference processes on NVIDIA GPUs. It breaks down VRAM usage by component, shows runtime and model details, and projects memory savings from quantization strategies.

Google Developers Blog post about integrating DFlash, a diffusion-style speculative decoding framework, into the vLLM TPU ecosystem to improve LLM serving speed on TPU v5p.

A vLLM blog post explaining Semantic Router as an open serving-layer runtime for bounded micro-agents, with routing patterns like Confidence, Ratings, ReMoM, Fusion, and Workflows behind a single model API.

A public slide deck about vLLM-compile, a project focused on bringing compiler optimizations to LLM inference and speeding up torch.compile for vLLM workflows.