Vendor access guide
How to allow Claude and Perplexity crawlers
Anthropic and Perplexity publish multiple identities. Match each one to the activity you actually want to allow.
Allow search identities explicitly, then verify the firewall.
For search discovery, review Claude-SearchBot and PerplexityBot. Treat ClaudeBot as a separate model-development choice, and treat user-requested identities according to current vendor documentation rather than assuming ordinary crawler behavior.
Separate Anthropic’s identities
Anthropic documents Claude-SearchBot for search, ClaudeBot for model development, and Claude-User for certain user-directed requests. That separation lets a site owner write a deliberate policy instead of allowing or blocking “Claude” as one undifferentiated system.
- Claude-SearchBot: consider for AI search discovery.
- ClaudeBot: decide separately for model development.
- Claude-User: review as user-requested access.
Separate Perplexity’s identities
Perplexity documents PerplexityBot for automatic search-related crawling and Perplexity-User for requests initiated by a person. The automatic crawler belongs in robots.txt policy; user-requested access may be described differently by the vendor.
- PerplexityBot: automatic discovery for Perplexity search experiences.
- Perplexity-User: certain visits made in response to a user request.
Publish the intended search policy
The following example allows the two automatic search identities. Add separate rules only where your organization has made a separate model-development decision.
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Verify CDN and firewall behavior
Security products can challenge or rate-limit bots independently of robots.txt. Inspect logs, response codes, and challenge pages. If the vendor publishes IP verification, use it rather than trusting the user-agent string.
Test several representative URLs: the homepage, one guide, a deep linked page, the sitemap, and a deliberately private route. The expected result should be explicit for each.
Verification checklist
- Choose access separately for every documented identity.
- Update robots.txt without disturbing ordinary search rules.
- Purge the CDN cache for robots.txt.
- Check representative public pages for 200 responses and readable HTML.
- Review first-party vendor documentation again before future policy changes.