quantization

Shoehorn is a Rust library and CLI for fitting GGUF LLMs into available Mac VRAM by choosing a per-tensor mixed-precision quantization plan. It can probe usable GPU memory, account for KV cache and compute buffers, generate or use an imatrix, write a fitted GGUF, and launch llama.cpp inference.

AutoRound is an open-source quantization toolkit for LLMs and VLMs, focused on high-accuracy low-bit inference across CPU, XPU, CUDA, and multiple deployment backends.

Official implementation of LittleBit and LittleBit-2 for sub-1-bit LLM compression via latent factorization and quantization-aware training.

An open-source CLI for inspecting live LLM inference processes on NVIDIA GPUs. It breaks down VRAM usage by component, shows runtime and model details, and projects memory savings from quantization strategies.

A GitHub repository for a 7-node ESP32-S3 cluster that runs a small LLM using 1.58-bit BitNet-style quantization over an SPI daisy-chain.

UltraCompress is a Python-based compression tool for large language models. The repo describes lossless 5-bit transformer compression, verification via SHA-256, a CLI on PyPI, and published model packs on Hugging Face.

An open-source Llama2 inference engine written in x86 real-mode assembly that fits in 1277 bytes and can boot directly from disk before any OS loads.