benchmark

A pure C runtime for Gemma 4 E2B CPU inference, with export, benchmarking, and numerical validation tools included.

An open public experiment that tests whether AI agents can complete everyday web errands, publishes the full transcripts, and scores 113 sites for agent-readiness.

ReactBench is a benchmark for evaluating coding agents on realistic React work, with published scores, cost comparisons, and example tasks focused on production-grade frontend issues.

An open robot football league where frontier AI models and open-source club code manage simulated Unitree G1 humanoids in daily matches, with live streams, match logs, and a public engine for teams to join.

Open-source causal-graph memory for AI agents with MCP support, scoring, active forgetting, and benchmark results published in the repo.

An open benchmark and leaderboard for AI CAD agents, with 308 prompts across 20 categories and layered scoring for geometry, engineering, manufacturability, and cognition.

A benchmark for coding agents that tests whether they can formally verify properties of real-world Rust web apps in Lean 4. It includes upstream code, i5h ports, extracted Lean models, proof attempts, and task-by-task results.

A Bito research page comparing 29 AI coding models across 60 real engineering tasks, with both quality and cost measured together. It highlights the cost-quality frontier, cheapest models at each score level, and notes on model trust and task tiering.

A benchmark page that aggregates public model lists from AI gateways into one matrix, showing which gateways offer each model and their per-million-token pricing.

Open-source repository for a NanoGPT training speedrun, focused on getting a 124M model to target loss on 8x H100 GPUs in under 90 seconds. The README includes run commands, Docker instructions, and a detailed list of model and systems optimizations.

A visual benchmark that asks AI coding agents to generate a single-file Three.js animation of a banana plant’s full life cycle, then compares the live results side by side.

A GitHub repo benchmarking TypeSafe’s Jev against Sonnet, GPT-5 nano, and local models on 770 Reddit AITA verdicts, with Brier scores, latency, and cost comparisons.

ObserverBench is an interactive benchmark and guided demo for testing whether internal AI monitoring methods help make safer decisions, not just better predictions. It includes real-model results, benchmark tasks, and a no-login walkthrough showing how warning scores affect review and harm that slips through.

Open-source harness for measuring how consistently hosted LLM APIs return the same output across repeated runs and over time, with raw transcripts, scoring, and a static leaderboard generator.

Open-source benchmark that measures which developer tools LLMs recommend when asked to build real web apps. It runs fixed prompts against a panel of models, parses the tool recommendations, and publishes rankings and head-to-head comparisons.

Public benchmark for comparing agent memory systems, with separate textual and coding tracks, submission flow, documentation, and API guidance.

An open-source benchmark for testing whether LLMs can follow long arithmetic derivations without using tools. It compares models on recall-proof inputs, logs digit accuracy, and reports results across multiple endpoints and model families.

DeepSWE is a benchmark for measuring frontier coding agents on original, long-horizon software engineering tasks. The page shows a leaderboard, methodology overview, task examples, and a full blog explaining the benchmark design and results.

A GitHub repository that publishes a week of controlled runs comparing seven Google models on the same software task, with transcripts, telemetry, screenshots, and scored benchmark artifacts.

A black-box benchmark report on how AI-generated tests detect functional bugs in live APIs across 20 scenarios and 7 systems.

A CUDA C++ project for training a tiny byte-level GPT quickly, with build, run, checkpoint, logging, and test instructions in the repo README.

A GitHub repo section documenting a benchmark for Vestige, a local-first Rust MCP memory layer for multi-agent coding fleets. The page explains the benchmark setup, what is being measured, how to reproduce the main claim, and where to find transcripts, evidence, and harness code.

A workbench report comparing MiniMax M3 and GLM 5.2 on autonomous coding tasks, with scored results, latency and cost data, task-type breakdowns, and examples of where each model performed better.

A showcase of 27 explorable Three.js worlds built autonomously by 9 coding models from the same brief.