pytorch

Open-source repository for a NanoGPT training speedrun, focused on getting a 124M model to target loss on 8x H100 GPUs in under 90 seconds. The README includes run commands, Docker instructions, and a detailed list of model and systems optimizations.

An open-source reinforcement learning training loop for LLMs, spanning small CPU-friendly experiments like CartPole to async RLVR runs on GPU clusters. The repo highlights a compact codebase, quickstart commands, task customization, and support for algorithms like REINFORCE, PPO, GRPO, and RLOO.

Official implementation of LittleBit and LittleBit-2 for sub-1-bit LLM compression via latent factorization and quantization-aware training.

A hands-on workshop repository that walks developers through building a small GPT-style language model from scratch, including tokenization, transformer blocks, training, and text generation.

A PyTorch reference implementation of DeepSeek Sparse Attention from the LLMs-from-scratch project, including a lightweight indexer, top-K token selection, and a GPT-style model with KV-cache support.

HyperSAE is a Python package for mechanistic interpretability that trains hyperbolic sparse autoencoders on LLM activations. The page includes installation steps, a quickstart example, benchmark tables, and a short software architecture overview.

A closed-loop controller for LLM training that adapts batch size, learning rate, checkpointing, and pruning to reduce OOM crashes on limited VRAM hardware.

A PyTorch research code file for adaptive topological control in latent spaces, with sphere-to-torus morphing, differentiable gating, and several geometry-inspired loss terms.

A research-oriented optimizer project claiming to cut AdamW optimizer-state memory for LLM training by 50% through factorized variance and frequency-domain compression. The repo includes a Python POC, benchmark claims, and a stated drop-in usage pattern for PyTorch training.