gpu

Release page for vLLM v0.28.0, a high-throughput and memory-efficient inference and serving engine for LLMs. The snapshot shows release highlights, model support updates, breaking changes, and installable artifacts for PyPI, Docker, ROCm, CPU, and XPU.

AirLLM is an open-source inference library that aims to run very large language models on small GPUs by decomposing models layer-wise and streaming them during execution. The snapshot shows a detailed README with quickstart code, model examples, supported model notes, and recent update history.

A web calculator for estimating whether local LLMs fit on specific hardware and how fast they may run. It covers VRAM, token throughput, latency, power, and cost across multiple GPUs and devices.

A technical primer that explains how LLM inference works, covering weights in GPU memory, prefill vs. decode, the KV cache, and batching.

Tokenstead helps people match AI models to their own hardware, with support for Macs, GPUs, and custom rigs. It shows which models can run locally, estimates speed, compares cloud pricing, and includes a source-cited tracker of organizations using different models.

AutoGPU is an open-source project for using AI agents to design and harden a GPU-like chip end to end, from Verilog through synthesis and place-and-route to GDSII. The repo shows a concrete fp8 matmul accelerator flow, quick-start steps, and documentation for running the design/hardening process.

An open-source handbook for production LLM serving and inference at scale, covering GPU fundamentals, KV cache, batching, quantization, speculative decoding, and engines like vLLM, SGLang, and TensorRT-LLM.

An out-of-tree vLLM plugin that compresses KV cache with STAR-KV to fit more concurrent requests and longer contexts on the same GPU memory budget.

Open-source AI infrastructure project focused on GPU memory management and serving, with a Python control plane, C++ data plane, and optional CUDA kernels.