testing

Understudy is a scenario-driven testing framework for AI agents. It simulates multi-turn users, records traces of messages and tool calls, and evaluates agent behavior with deterministic checks, optional LLM judges, and reports.

Excalibur is an open-source AI coding agent for product engineers. It runs from the terminal, supports multiple model providers, and covers the full workflow from discovery and planning through implementation, verification, review, and shipping.

An open-source CLI for testing LLM prompts across consistency, latency, cost, and behavior, with HTML reports and support for OpenAI and Anthropic.

Python package for recording real LLM API responses and replaying them in tests so LLM-driven code can run deterministically without live API calls.

CLI for testing and improving documentation with simulated developer agents. It checks whether docs are clear enough for agents to complete real tasks, reports where they get stuck, and can generate proposed edits.

Weir is a CI gate for AI agents that reads OpenTelemetry traces and fails builds when sensitive data reaches forbidden sinks. It includes a gauge for trace coverage, taint tracking, HTML reports, and rule-based scans for agent workflows.

Backprompter is a workspace for creating, testing, and deploying AI agents without backend code. The page shows prompt-based agent setup, simulated users and evaluations, hosted chat UI deployment, SDK/API access, RAG, voice chat, and Stripe billing.

SHIP is an agentic engineering platform that assigns issues to specialist AI agents to plan, build, review, deploy, and test software on your own infrastructure. The page shows issue-to-PR workflow, mission timelines, cost tracking, model selection, and support for GitHub and Linear.

An open-source tool that watches Playwright failures, inspects the live page, and opens a GitHub PR with a proposed fix.

Omnideck describes a self-testing browser-tool workflow where an AI agent runs daily checks on real websites, diagnoses failures against its own source code, and files GitHub issues with proposed fixes.

An open-source, self-hosted AI DevOps pipeline that ingests GitHub webhooks, generates code patches with an LLM, runs them in network-isolated Docker sandboxes, and posts validated fixes back as PR comments.

A Copilot skill suite that enforces structured coding workflows for AI-assisted development, with design, planning, TDD, debugging, verification, and cleanup steps plus a live task-graph visualization server.

An open-source enforcement layer for AI coding agents that adds frozen specs, tamper-detected tests, independent verification, and hard-blocking gates so unverified work cannot pass.

A tiny single-tasking DOS clone written from scratch in x86 assembly, built to boot and run period games in emulators. The repo includes a detailed README, build instructions, docs, and automated QEMU regression testing.

A source-available JavaScript browser agent for asserting end-to-end behavior in real user sessions. The repo explains the annotation-based approach, how it runs in staging and production, and how to install and initialize it via CDN or npm.

A GitHub repo for evaluating agentic AI pipeline systems, with guidance for defining metrics, building eval cases, running repeatable tests, and tracking regressions.

Pure Effect is a small JavaScript library for building testable business logic as data-driven commands instead of relying on mocks. It includes primitives like Success, Failure, Command, Ask, Retry, and Parallel, plus helpers like effectPipe and runEffect for composing and executing flows.