AI Crawler & Bot Directory
Directory of 28 AI crawlers with robots.txt rules to allow or block each one
1 ratings4.21
Saves2
CategoryDeveloper Tools
PlatformWeb
PricingFree
A maintained directory listing 28 AI web crawlers used by major AI models and services, providing detailed profiles, robots.txt compliance information, and copy-paste rules to control crawler access. Includes tools like an AI Crawler Access Checker and robots.txt generator to manage crawler permissions.
ai
Ratings & Reviews
Write a reviewBe the first to share a thoughtful review.
Information
CategoryDeveloper Tools
PlatformsWeb
PricingFree
AccountNot confirmed
ListedSeptember 2026
WebVisit
Report a problem
Website checks
Mobile PageSpeed
Automated checks of the public website by Google Lighthouse 13.5.0. Scores can vary between runs.
Checked Oct 27, 2026
Test this website on PageSpeedWebsite checks refresh automatically each week.
- Performance
- 95 out of 100, Good
- Accessibility
- 100 out of 100, Good
- Best Practices
- 100 out of 100, Good
- SEO
- 100 out of 100, Good
Around the Web
Show HN2 source points · 0 comments