Policy patterns
AI crawler robots.txt examples
The safest robots.txt file is the smallest one that accurately expresses a reviewed policy.
Choose the policy first; copy the example second.
Use crawler-specific groups when you want different rules for AI search and model development. Preserve existing search-engine and private-path rules, publish the file at the canonical hostname, and test the exact production response.
Allow AI search and block model development
This pattern keeps documented search-discovery crawlers open while declining several model-development crawlers. Confirm every token against the vendor’s current documentation before deployment.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /Allow documented crawlers on public content
If the site owner has chosen broad public access, explicit allow rules can make that decision easy to audit. They are usually unnecessary when no broader rule blocks access, but clarity can be useful in a complex file.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Keep non-public paths out of crawler policy
Robots.txt is public and advisory. Use authentication and authorization for private information. A disallow rule can reduce crawling, but it does not make a URL secret.
User-agent: *
Disallow: /wp-admin/
Disallow: /account/
Disallow: /checkout/Watch for: Do not put sensitive path names in robots.txt unless exposing those names is acceptable. Protect the underlying resource at the application layer.
Avoid common deployment errors
One missing slash or an inherited wildcard group can change the result. Caches and CDNs can also serve an older file after WordPress shows the new setting.
- Do not paste a second policy without reconciling the existing
User-agent: *group. - Do not block CSS or JavaScript needed to understand the public page.
- Do not assume a WordPress editor preview matches the public
/robots.txt. - Do not promise indexing or citations merely because access is allowed.
Verification checklist
- Save the current robots.txt before editing.
- Validate every user-agent token against first-party documentation.
- Fetch the public file through the CDN after deployment.
- Test representative allowed and blocked paths.
- Confirm the sitemap line still points to the canonical sitemap.