benchmarking

SigMap is a deterministic grounding layer for AI code work. It generates a signature-and-evidence map, helps pick relevant files, validates coverage, and judges whether an AI answer is grounded in the repo.

BoundaryBench is an open-source benchmark for coding agents running under hardened sandbox policies. It compares harnesses like Claude Code, Codex, Terminus 2, and Grok on Terminal-Bench tasks and includes quickstart, policy controls, and result export tools.

Open-source infrastructure for computer-use agents, with sandboxes, SDKs, benchmarks, and desktop automation tooling for macOS, Linux, Windows, and Android. The repo also includes Cua Driver, CuaBot, Cua-Bench, and Lume for VM management.

오픈소스 AI 인프라 산정 도구로, 보유 GPU나 목표 워크로드를 입력해 실행 가능한 모델, 필요 장비, 배치안, TCO 비교까지 계산합니다. LLM, RAG, VLM, 아바타, 음성 워크로드를 다루며 한국어/영어 UI와 여러 추천 경로를 제공합니다.

A community-tracked index of how users feel AI model quality is changing, updated hourly from public feedback and direct submissions.

An interactive guide for choosing local AI hardware and matching open-weights LLMs to specific builds. It offers budget-based build recommendations, a hardware picker, model-to-hardware compatibility, and a state-of-the-local-AI snapshot.

A TypeScript CLI and SDK for testing whether Agent Skills improve model outputs by running with-skill vs baseline evaluations and generating reports.

A loadable SQLite extension that adds column-oriented storage and analytics for fast local OLAP-style queries, with benchmark data and build instructions.

A research article from Applied Compute on how agentic, tool-using workloads differ from traditional LLM benchmarks, with production observations, workload profiles, and an open-source harness for replaying traces.

A research repo for reducing hallucinations in LLM-generated code using semantic triangulation, with setup, benchmarking, experimentation, and reproducibility instructions.

Open-source runtime for coding agents that sits underneath Claude Code to reduce tool calls, shorten context, and track savings. The page includes installation steps, a local savings check, and benchmark results comparing cost and performance.

A visual benchmark that asks AI coding agents to generate a single-file Three.js animation of a banana plant’s full life cycle, then compares the live results side by side.

Caplets is a developer tool that wraps MCP servers into smaller capability-based surfaces for coding agents. The site explains the workflow, setup, example capabilities like OSV, GitHub, and Sourcegraph, and includes benchmark results plus docs links.

A persistent memory layer for AI agents, built as a standard MCP server with local setup, dashboard, and benchmark claims against other memory tools.

arXiv paper describing QUEST, an open family of deep research agents from 2B to 35B parameters, plus a synthetic-task training recipe and released models, data, and scripts.

Experimental page-aware Metal runtime and reproducibility harness for running oversized sparse mixture-of-experts models on Apple Silicon. The repo documents GPT-OSS 120B support, performance results on 64 GiB and 16 GiB machines, and the runtime/harness structure.

A local Rust CLI for A/B testing two CLAUDE.md files against a fixed SWE-bench Lite smoke set, with doctor, run, and report commands.

A research page comparing 153 autonomous runs across 18 frontier models on the nanoGPT optimizer speedrun, with rankings, trajectories, token/compute stats, and equal-budget comparisons.

Reference implementation for using Linux Pressure Stall Information to trim an LLM KV cache under memory pressure. The repo includes requirements, basic usage commands, a simulator, a llama.cpp runner, and benchmark scripts with example results.

A GitHub release for Microsoft’s Agent Lightning Skill, which helps coding agents optimize other AI agents using benchmarks and iterative improvements across prompts, tools, workflows, models, and reasoning settings.

A Python CLI and research toolkit for simulating AI training math before model training. It estimates memory, training time, token budget, config risks, benchmark speed, and reliability, with optional native Rust and C backends.

An Apple Silicon–optimized inference build of Bonsai 1.7B with custom Metal kernels, benchmark results, quick-start instructions, and a bundled OpenAI-compatible server.

A blog post about verifiable RAG that benchmarks open-source NLI verifiers against Claude on RAGTruth and describes a Python library for sentence-level citation and claim verification.

A blog post from Augment Code comparing its coding agent, Auggie, against Claude Code on Opus 4.7. It presents benchmark results, token usage, cost comparisons, and an explanation of the Context Engine and Prism router.