benchmarking

SigMap is a deterministic grounding layer for AI code work. It generates a signature-and-evidence map, helps pick relevant files, validates coverage, and judges whether an AI answer is grounded in the repo.

BoundaryBench is an open-source benchmark for coding agents running under hardened sandbox policies. It compares harnesses like Claude Code, Codex, Terminus 2, and Grok on Terminal-Bench tasks and includes quickstart, policy controls, and result export tools.

Open-source infrastructure for computer-use agents, with sandboxes, SDKs, benchmarks, and desktop automation tooling for macOS, Linux, Windows, and Android. The repo also includes Cua Driver, CuaBot, Cua-Bench, and Lume for VM management.

A browser-based lab for running, comparing, and benchmarking small language models on-device with WebGPU.

Free curriculum for learning AI agents, LLM evaluation, and benchmarking, with 42 lessons, interactive labs, runnable JavaScript, and saved progress.

Technical research page for BlueVetaUpright, a product-photo orientation correction system. The page explains the problem, the system design, dataset methodology, and benchmark results, including comparisons against ChatGPT, Gemini, and Claude.

오픈소스 AI 인프라 산정 도구로, 보유 GPU나 목표 워크로드를 입력해 실행 가능한 모델, 필요 장비, 배치안, TCO 비교까지 계산합니다. LLM, RAG, VLM, 아바타, 음성 워크로드를 다루며 한국어/영어 UI와 여러 추천 경로를 제공합니다.

A community-tracked index of how users feel AI model quality is changing, updated hourly from public feedback and direct submissions.

An interactive guide for choosing local AI hardware and matching open-weights LLMs to specific builds. It offers budget-based build recommendations, a hardware picker, model-to-hardware compatibility, and a state-of-the-local-AI snapshot.

A TypeScript CLI and SDK for testing whether Agent Skills improve model outputs by running with-skill vs baseline evaluations and generating reports.

DrivingBench is a benchmark that tests frontier LLMs on real-world driving tasks in a Toyota Corolla with a cone course, leaderboard, run traces, and replay videos.

Ante is a self-contained coding agent for the terminal with local model support, multi-agent orchestration, and programmatic server modes. The docs highlight offline operation, low resource use, public benchmarks, and integrations for providers, MCP, Slack/Discord, and custom engines.

A loadable SQLite extension that adds column-oriented storage and analytics for fast local OLAP-style queries, with benchmark data and build instructions.

A research article from Applied Compute on how agentic, tool-using workloads differ from traditional LLM benchmarks, with production observations, workload profiles, and an open-source harness for replaying traces.

Open-source code search for coding agents, built on Jev, with benchmarked speed and token-use gains across Codex, OpenCode, and Claude Code.

A benchmark page that aggregates public model lists from AI gateways into one matrix, showing which gateways offer each model and their per-million-token pricing.

Anypick is an open library for selecting the best LLM across providers using filters for capabilities, pricing, and benchmarks. It includes Python and TypeScript implementations, provider support for OpenRouter and Vercel AI Gateway, and examples showing how to pick a model by constraints and strategy.

A research repo for reducing hallucinations in LLM-generated code using semantic triangulation, with setup, benchmarking, experimentation, and reproducibility instructions.

Open-source runtime for coding agents that sits underneath Claude Code to reduce tool calls, shorten context, and track savings. The page includes installation steps, a local savings check, and benchmark results comparing cost and performance.

A visual benchmark that asks AI coding agents to generate a single-file Three.js animation of a banana plant’s full life cycle, then compares the live results side by side.

A LangGraph-based workflow that generates, benchmarks, validates, and iterates CUDA kernels using a C++/CUDA harness. It targets workload-specific optimization with support for NVIDIA research lookup and optional Nsight Compute profiling.

A utility for AMD GPU users running local AI on Windows or Linux. It detects the GPU, applies the right HSA_OVERRIDE_GFX_VERSION setting, offers health checks, and benchmarks Vulkan vs. HIP/ROCm for local LLM setups.

A benchmark writeup comparing GraphQL-over-MCP and REST-over-MCP setups, showing lower token use and inference cost for GraphQL in agentic tasks.

A detailed post from pact0 about testing whether blank AI agents could complete a real product workflow, what failed, and the fixes they shipped after the audit.