openai-compatible-api

WebLLM is a high-performance in-browser LLM inference engine that runs locally in the browser with WebGPU acceleration. It exposes an OpenAI-compatible API, supports streaming and JSON mode, and includes examples for building chat apps and browser extensions.

AURA is a Rust-based SRE agent platform for investigating and fixing production incidents. The repository shows a quick start, production-safety controls, multi-model support, MCP and RAG integrations, and an example incident workflow using traces, logs, and Kubernetes history.

Open source inference engine for agents that self-tunes on the user’s hardware to run open models faster, with desktop app support and one-click connections to popular agents.

An open-source mixture-of-agents server that fans prompts out to multiple LLMs, judges the responses, and synthesizes a final answer. It exposes an OpenAI-compatible API and also includes an agentic coding harness plus integrations for tools like Claude Code, aider, Continue, LangChain, and Pi.

GIDE is a privacy-first AI coding command center that runs local or cloud models in a code editor and terminal. The page shows live session examples, a write-approval gate, CLI support, and an OpenAI-compatible local API.

Open-source self-hosted inference server for agent workloads, with an OpenAI-compatible API, multi-model serving, and production cluster tooling.

Wyolet Relay is a self-hosted LLM router that exposes OpenAI- and Anthropic-compatible endpoints in front of multiple providers. It pools API keys, handles failover and rate limits, tracks usage and cost, and includes a quickstart plus Docker-based deployment.

A pure-Python LLM inference engine and server with CUDA/HIP/Metal code generation, OpenAI-compatible API support, built-in RAG, and multi-GPU backend support.

A macOS command-line tool for on-device AI and voice tasks, including speak, listen, transcribe, chat, and an OpenAI-compatible local server.

Flux is a Rust-based LLM inference planner and runtime that measures local hardware, chooses a model placement plan, and serves models behind an OpenAI-compatible API.

A detailed walkthrough for running a local coding agent on macOS with llama.cpp, Gemma 4, MTP speculative decoding, image support, and Pi as the agent interface.

An Apple Silicon–optimized inference build of Bonsai 1.7B with custom Metal kernels, benchmark results, quick-start instructions, and a bundled OpenAI-compatible server.

A vLLM blog post explaining Semantic Router as an open serving-layer runtime for bounded micro-agents, with routing patterns like Confidence, Ratings, ReMoM, Fusion, and Workflows behind a single model API.