quantization

AutoRound is an open-source quantization toolkit for LLMs and VLMs, focused on high-accuracy low-bit inference across CPU, XPU, CUDA, and multiple deployment backends.

An open-source CLI for inspecting live LLM inference processes on NVIDIA GPUs. It breaks down VRAM usage by component, shows runtime and model details, and projects memory savings from quantization strategies.

UltraCompress is a Python-based compression tool for large language models. The repo describes lossless 5-bit transformer compression, verification via SHA-256, a CLI on PyPI, and published model packs on Hugging Face.

An open-source Llama2 inference engine written in x86 real-mode assembly that fits in 1277 bytes and can boot directly from disk before any OS loads.