LLM evaluation

Focal Prompt is an analysis lab for studying how prompts and prompt sections influence model behavior. It includes focus labeling, prospective and retrospective focus prediction, baseline output sampling, ablation tests, batch analysis, and an agent builder.

Open-source benchmark that measures which developer tools LLMs recommend when asked to build real web apps. It runs fixed prompts against a panel of models, parses the tool recommendations, and publishes rankings and head-to-head comparisons.

Crilio is a Python CLI and CI gate for testing AI prompts and catching regressions before they ship. It versions prompt tests in crilio.yaml, runs model calls, uses a judge model to score responses, and can block pull requests in GitHub Actions when checks fail.

An open-source benchmark for testing whether LLMs can follow long arithmetic derivations without using tools. It compares models on recall-proof inputs, logs digit accuracy, and reports results across multiple endpoints and model families.

An open-source research framework for structured LLM evaluation, claim verification, and source-grounded reflective reasoning. The repo describes modular components for retrieval, semantic scoring, skeptical claim checking, and benchmark-style epistemic assessment.

AdvertBench is a web app for ranking AI-generated image ad sets with Elo voting. The page shows head-to-head ad comparisons, a leaderboard, and a sample prompt for generating ads, making the product purpose easy to understand.

A research preprint and dataset on how five frontier LLMs disagree when judging 1,000 real-world fact-check claims, with accompanying corpus, raw results, and code repository.

Pull request adding TY25 support for moonshotai/kimi-k3 in TaxCalcBench through OpenRouter. The snapshot shows benchmark wiring, PDF handling details, saved outputs and evaluation reports, and regenerated charts/results for the full TY25 set.

Giskard presents an AI red-teaming and continuous evaluation platform focused on finding hallucinations, security issues, and agent vulnerabilities. The page explains how it compares tools for agent-level testing, regression checks, and guardrail workflows.