llama.cpp

Shoehorn is a Rust library and CLI for fitting GGUF LLMs into available Mac VRAM by choosing a per-tensor mixed-precision quantization plan. It can probe usable GPU memory, account for KV cache and compute buffers, generate or use an imatrix, write a fitted GGUF, and launch llama.cpp inference.

A minimalist, terminal-native coding agent written in C. It runs as a single native binary, supports local models through llama.cpp and Ollama, and can also connect to OpenAI, Anthropic, OpenRouter, and Codex.

A lightweight API key proxy for self-hosted llama.cpp servers with per-user token usage tracking, rate limiting, and multi-server routing.

Wizard is a self-extending autonomous agent shipped as a single Rust binary. It supports local and hosted models, live provider switching, MCP tools, self-modification, a messaging gateway, and built-in benchmarking.

SnapName is a macOS app that watches a screenshot or image folder and uses bundled local AI to suggest better filenames without uploading images.

A llama.cpp fork that adds a JEV-compatible decision API for Choice, Score, and Noul evaluations while keeping standard LLM inference support.

A lightweight Flask-based LLM proxy that forwards Ollama, OpenAI, and llama.cpp-style requests to NVIDIA’s OpenAI-compatible API, with streaming support, retries, caching, auth, and usage stats.

Reference implementation for using Linux Pressure Stall Information to trim an LLM KV cache under memory pressure. The repo includes requirements, basic usage commands, a simulator, a llama.cpp runner, and benchmark scripts with example results.

Galahad is an AI memory layer that saves and restores model KV cache, supports document recall, and plugs into vLLM, SGLang, and llama.cpp. The page also shows agent features like replay, branching, rewind, and fault-finding.

A detailed walkthrough for running a local coding agent on macOS with llama.cpp, Gemma 4, MTP speculative decoding, image support, and Pi as the agent interface.

A local AI coding harness that turns business-rule specs into a constrained JSON AST, validates it, and compiles it into Clojure code while logging generation and repair artifacts.

A GitHub repository for a custom llama.cpp backend that runs Qwen3-0.6B GGUF models on an Axera AX8850 NPU accelerator, with build/run notes, environment flags, architecture details, and test status.

A GitHub repository for an engine-agnostic KV cache fragment system for on-device LLM inference, with Android and Go components, adapters for llama.cpp/MLC-LLM/ONNX Runtime, and benchmark and monitoring code.

A GitHub repository for a latency-separated AI memory retrieval and RAG system. The README describes fetch, compute, and ANN search stages, includes benchmark ranges, and exposes a public test endpoint.