llm-proxy

An OpenAI-compatible LLM proxy that routes requests to the fastest healthy provider. It runs as a small sidecar, supports latency/error-aware routing, session affinity, and Vertex AI, and can be deployed as a single static binary or Docker container.

LiteLLM is moving its AI gateway to Rust and claims major gains in throughput, memory use, and request overhead while keeping the same config, API, and provider coverage.