Research / AI policy / web crawler intelligence
A data-backed study showing which major websites block specific AI crawlers in robots.txt, with breakdowns by crawler, site type, and a downloadable CSV.
The page clearly presents a real, AI-adjacent product and research offering: a crawl-based study of 945 websites, monthly rechecks, downloadable data, and live site checking. It gives enough visible detail for a useful directory listing, including methodology, key findings, crawler lists, and adjacent monitoring tools.
The page is strong on findings, but it is not fully clear how fresh the dataset is beyond the stated check date, and there is limited detail on the exact coverage of the monthly monitoring workflow or the full contents of the CSV.
Last evaluated 6 hours ago. Current rank #609. Up 2 spots in the rankings.
Primus is an autonomous AI researcher that hypothesizes, reads papers, writes code, runs experiments on compute, and drafts research papers. The page shows example tasks, published-paper claims, waitlist access, and positioning for ML research workflows.
An interactive dashboard that analyzes New York Times coverage since 2000 using the NYT Archive API, with views for reporters, beats, sections, subjects, geography, obituaries, and corrections.
A research article benchmarking bot detection systems against scripted browsers, stealth browsers, and modern AI browsing agents, with comparative results across five detection products.
An open benchmark and leaderboard for AI CAD agents, with 308 prompts across 20 categories and layered scoring for geometry, engineering, manufacturability, and cognition.
