inference

A pure C runtime for Gemma 4 E2B CPU inference, with export, benchmarking, and numerical validation tools included.

Open source inference engine for agents that self-tunes on the user’s hardware to run open models faster, with desktop app support and one-click connections to popular agents.

Release page for vLLM v0.28.0, a high-throughput and memory-efficient inference and serving engine for LLMs. The snapshot shows release highlights, model support updates, breaking changes, and installable artifacts for PyPI, Docker, ROCm, CPU, and XPU.

AirLLM is an open-source inference library that aims to run very large language models on small GPUs by decomposing models layer-wise and streaming them during execution. The snapshot shows a detailed README with quickstart code, model examples, supported model notes, and recent update history.

AutoRound is an open-source quantization toolkit for LLMs and VLMs, focused on high-accuracy low-bit inference across CPU, XPU, CUDA, and multiple deployment backends.

OpenTPU is an open-source AI accelerator project with RTL, ISA, simulator, compiler, profiler, and host software in one repo. The page shows it running real models like Qwen and LFM on a Kintex-7 PCIe card, with detailed performance results and documentation links.

A web calculator for estimating whether local LLMs fit on specific hardware and how fast they may run. It covers VRAM, token throughput, latency, power, and cost across multiple GPUs and devices.

A Swedish AI inference API offering GLM 5.1 on owned H200 hardware in Stockholm, with OpenAI-compatible endpoints, SEK pricing, and an emphasis on EU data residency.

A pure C implementation for training and running a tiny GPT, with build instructions, sample output, and performance notes for macOS, Linux, and Windows.

A technical primer that explains how LLM inference works, covering weights in GPU memory, prefill vs. decode, the KV cache, and batching.

A job board focused on AI engineering roles, with first-party listings for RAG, agents, evals, inference, and MCP-related work. The page shows live filters, location and seniority breakdowns, and a nightly refreshed feed of open roles.

An open-source handbook for production LLM serving and inference at scale, covering GPU fundamentals, KV cache, batching, quantization, speculative decoding, and engines like vLLM, SGLang, and TensorRT-LLM.

Flux is a Rust-based LLM inference planner and runtime that measures local hardware, chooses a model placement plan, and serves models behind an OpenAI-compatible API.

A curated GitHub resource list for learning AI performance engineering, covering GPU fundamentals, kernel optimization, inference engines, and distributed inference.

Uno is a research repository for speeding up LLM inference with discrete diffusion and lossless multi-token decoding. The repo includes inference, training, and evaluation code, plus installation steps, checkpoints, and example workflows.

Self-hosted, Kubernetes-native platform for distributed multi-model LLM inference. It includes an OpenAI-compatible API, installer flow, and infrastructure-as-code deployment with support for air-gapped environments.

Scalattice is an inference API and hosting platform that lets developers point the OpenAI SDK at its OpenAI-compatible endpoint with minimal code changes. The page includes Python and curl examples, model IDs, pricing references, and basic troubleshooting.

A pull request adding AMD GPU support to tiny-vLLM through ROCm/HIP while keeping the existing CUDA build path unchanged. The snapshot describes the compatibility header, CMake option, architecture selection, and validation on multiple AMD GPUs.

An Apple Silicon–optimized inference build of Bonsai 1.7B with custom Metal kernels, benchmark results, quick-start instructions, and a bundled OpenAI-compatible server.

A GitHub repository for a custom llama.cpp backend that runs Qwen3-0.6B GGUF models on an Axera AX8850 NPU accelerator, with build/run notes, environment flags, architecture details, and test status.

An interactive simulation about how inference providers ration access to stronger LLMs when demand spikes. Users can compare the default industry rule with their own policy and explore who gets the strong model.