llm-inference

Open-source C++ and CUDA LLM inference engine inspired by vLLM, with a teaching-focused course that walks through model serving, batching, KV cache, and attention kernels.

A research article from Applied Compute on how agentic, tool-using workloads differ from traditional LLM benchmarks, with production observations, workload profiles, and an open-source harness for replaying traces.

A pure-Python LLM inference engine and server with CUDA/HIP/Metal code generation, OpenAI-compatible API support, built-in RAG, and multi-GPU backend support.

Otlet is a Postgres extension for running local LLM inference next to database rows. The README shows SQL examples, model registration, task execution, and an entity-resolution workflow with typed outputs.

Bootable Rust no_std kernel built as an inference substrate for LLMs, with tensor-native memory allocation, layer-boundary scheduling, and streaming-focused runtime primitives.

An open-source CLI for inspecting live LLM inference processes on NVIDIA GPUs. It breaks down VRAM usage by component, shows runtime and model details, and projects memory savings from quantization strategies.

Experimental page-aware Metal runtime and reproducibility harness for running oversized sparse mixture-of-experts models on Apple Silicon. The repo documents GPT-OSS 120B support, performance results on 64 GiB and 16 GiB machines, and the runtime/harness structure.

A technical demo that streams Llama-2 weights over Wi‑Fi to run inference on an ESP32 with limited RAM. The repo explains the approach, quickstart steps, performance notes, and the split between ESP32 firmware and a PC host process.

A GitHub repository guide to running state-of-the-art LLMs on local hardware, with concrete build notes, cost breakdowns, and runnable configs for large models.

Google Developers Blog post about integrating DFlash, a diffusion-style speculative decoding framework, into the vLLM TPU ecosystem to improve LLM serving speed on TPU v5p.

A DeepSeek paper about DSpark, a full-stack codebase for training and evaluating speculative decoding algorithms to speed up LLM inference.

A technical blog post from the Cua repository describing a process-scoped Metal capability shim for macOS VMs that improves llama.cpp performance on Apple Silicon, with benchmark results and reproduction details.

A public slide deck about vLLM-compile, a project focused on bringing compiler optimizations to LLM inference and speeding up torch.compile for vLLM workflows.