openai-compatible-api

WebLLM is a high-performance in-browser LLM inference engine that runs locally in the browser with WebGPU acceleration. It exposes an OpenAI-compatible API, supports streaming and JSON mode, and includes examples for building chat apps and browser extensions.

A pure-Python LLM inference engine and server with CUDA/HIP/Metal code generation, OpenAI-compatible API support, built-in RAG, and multi-GPU backend support.

An Apple Silicon–optimized inference build of Bonsai 1.7B with custom Metal kernels, benchmark results, quick-start instructions, and a bundled OpenAI-compatible server.