llm-inference

Open-source C++ and CUDA LLM inference engine inspired by vLLM, with a teaching-focused course that walks through model serving, batching, KV cache, and attention kernels.

A research article from Applied Compute on how agentic, tool-using workloads differ from traditional LLM benchmarks, with production observations, workload profiles, and an open-source harness for replaying traces.

AI inference for the JVM with chat, vision, embeddings, and text-to-speech, plus Java-first building blocks and example snippets.

A pure-Python LLM inference engine and server with CUDA/HIP/Metal code generation, OpenAI-compatible API support, built-in RAG, and multi-GPU backend support.

Otlet is a Postgres extension for running local LLM inference next to database rows. The README shows SQL examples, model registration, task execution, and an entity-resolution workflow with typed outputs.

Bootable Rust no_std kernel built as an inference substrate for LLMs, with tensor-native memory allocation, layer-boundary scheduling, and streaming-focused runtime primitives.

A llama.cpp fork that adds a JEV-compatible decision API for Choice, Score, and Noul evaluations while keeping standard LLM inference support.

An open-source CLI for inspecting live LLM inference processes on NVIDIA GPUs. It breaks down VRAM usage by component, shows runtime and model details, and projects memory savings from quantization strategies.

Experimental page-aware Metal runtime and reproducibility harness for running oversized sparse mixture-of-experts models on Apple Silicon. The repo documents GPT-OSS 120B support, performance results on 64 GiB and 16 GiB machines, and the runtime/harness structure.

A technical demo that streams Llama-2 weights over Wi‑Fi to run inference on an ESP32 with limited RAM. The repo explains the approach, quickstart steps, performance notes, and the split between ESP32 firmware and a PC host process.

A GitHub repository guide to running state-of-the-art LLMs on local hardware, with concrete build notes, cost breakdowns, and runnable configs for large models.

Open-source research workbench for fail-closed LLM inference routing based on measured GPU workloads. The repo documents Phase 1 results, reproducible experiments, quick-start commands, and a policy that only promotes exact measured workload buckets.

Google Developers Blog post about integrating DFlash, a diffusion-style speculative decoding framework, into the vLLM TPU ecosystem to improve LLM serving speed on TPU v5p.

A DeepSeek paper about DSpark, a full-stack codebase for training and evaluating speculative decoding algorithms to speed up LLM inference.

A technical blog post from the Cua repository describing a process-scoped Metal capability shim for macOS VMs that improves llama.cpp performance on Apple Silicon, with benchmark results and reproduction details.

A public slide deck about vLLM-compile, a project focused on bringing compiler optimizations to LLM inference and speeding up torch.compile for vLLM workflows.