llm-inference

Open-source C++ and CUDA LLM inference engine inspired by vLLM, with a teaching-focused course that walks through model serving, batching, KV cache, and attention kernels.

A research article from Applied Compute on how agentic, tool-using workloads differ from traditional LLM benchmarks, with production observations, workload profiles, and an open-source harness for replaying traces.

A pure-Python LLM inference engine and server with CUDA/HIP/Metal code generation, OpenAI-compatible API support, built-in RAG, and multi-GPU backend support.

Bootable Rust no_std kernel built as an inference substrate for LLMs, with tensor-native memory allocation, layer-boundary scheduling, and streaming-focused runtime primitives.

An open-source CLI for inspecting live LLM inference processes on NVIDIA GPUs. It breaks down VRAM usage by component, shows runtime and model details, and projects memory savings from quantization strategies.

A GitHub repository guide to running state-of-the-art LLMs on local hardware, with concrete build notes, cost breakdowns, and runnable configs for large models.

Google Developers Blog post about integrating DFlash, a diffusion-style speculative decoding framework, into the vLLM TPU ecosystem to improve LLM serving speed on TPU v5p.

A DeepSeek paper about DSpark, a full-stack codebase for training and evaluating speculative decoding algorithms to speed up LLM inference.

A public slide deck about vLLM-compile, a project focused on bringing compiler optimizations to LLM inference and speeding up torch.compile for vLLM workflows.