gguf

Shoehorn is a Rust library and CLI for fitting GGUF LLMs into available Mac VRAM by choosing a per-tensor mixed-precision quantization plan. It can probe usable GPU memory, account for KV cache and compute buffers, generate or use an imatrix, write a fitted GGUF, and launch llama.cpp inference.

Native Zig inference server for Apple Silicon Macs that runs local LLMs with MLX and GGUF support, plus OpenAI- and Anthropic-compatible APIs. The page highlights one-click local setup, Ollama compatibility, agent tools, MCP support, and performance claims against LM Studio.

An Apple Silicon–optimized inference build of Bonsai 1.7B with custom Metal kernels, benchmark results, quick-start instructions, and a bundled OpenAI-compatible server.

A GitHub repository for a custom llama.cpp backend that runs Qwen3-0.6B GGUF models on an Axera AX8850 NPU accelerator, with build/run notes, environment flags, architecture details, and test status.