openai-compatible-api

WebLLM is a high-performance in-browser LLM inference engine that runs locally in the browser with WebGPU acceleration. It exposes an OpenAI-compatible API, supports streaming and JSON mode, and includes examples for building chat apps and browser extensions.

An open-source mixture-of-agents server that fans prompts out to multiple LLMs, judges the responses, and synthesizes a final answer. It exposes an OpenAI-compatible API and also includes an agentic coding harness plus integrations for tools like Claude Code, aider, Continue, LangChain, and Pi.

Wyolet Relay is a self-hosted LLM router that exposes OpenAI- and Anthropic-compatible endpoints in front of multiple providers. It pools API keys, handles failover and rate limits, tracks usage and cost, and includes a quickstart plus Docker-based deployment.

A pure-Python LLM inference engine and server with CUDA/HIP/Metal code generation, OpenAI-compatible API support, built-in RAG, and multi-GPU backend support.

A macOS command-line tool for on-device AI and voice tasks, including speak, listen, transcribe, chat, and an OpenAI-compatible local server.

A detailed walkthrough for running a local coding agent on macOS with llama.cpp, Gemma 4, MTP speculative decoding, image support, and Pi as the agent interface.

An Apple Silicon–optimized inference build of Bonsai 1.7B with custom Metal kernels, benchmark results, quick-start instructions, and a bundled OpenAI-compatible server.

A vLLM blog post explaining Semantic Router as an open serving-layer runtime for bounded micro-agents, with routing patterns like Confidence, Ratings, ReMoM, Fusion, and Workflows behind a single model API.