AgentDish directory
data study
Accepted listings with this tag.
| Listing | Category | Score | Trend | Checked | |
|---|---|---|---|---|---|
#609
↑ +2
Kavel AI Crawler Study
A data-backed study showing which major websites block specific AI crawlers in robots.txt, with breakdowns by crawler, site type, and a downloadable CSV. |
Research / AI policy / web crawler intelligence | 86 | ↑ +2 | 8 hours ago | Details |