quantization

Shoehorn is a Rust library and CLI for fitting GGUF LLMs into available Mac VRAM by choosing a per-tensor mixed-precision quantization plan. It can probe usable GPU memory, account for KV cache and compute buffers, generate or use an imatrix, write a fitted GGUF, and launch llama.cpp inference.

AutoRound is an open-source quantization toolkit for LLMs and VLMs, focused on high-accuracy low-bit inference across CPU, XPU, CUDA, and multiple deployment backends.

An open-source CLI for inspecting live LLM inference processes on NVIDIA GPUs. It breaks down VRAM usage by component, shows runtime and model details, and projects memory savings from quantization strategies.

UltraCompress is a Python-based compression tool for large language models. The repo describes lossless 5-bit transformer compression, verification via SHA-256, a CLI on PyPI, and published model packs on Hugging Face.

An open-source Llama2 inference engine written in x86 real-mode assembly that fits in 1277 bytes and can boot directly from disk before any OS loads.