How to allow ChatGPT to crawl your website

Choose the access purpose

Start with the outcome. Search discovery, a visit requested by a user, and use for model development are different activities. A single blanket rule can accidentally block the feature you want or allow a use you meant to decline.

Write the policy down in plain language first: “Allow AI search, allow user-requested visits, and block model development,” for example. Then translate that decision into crawler-specific controls.

• OAI-SearchBot: automatic crawling associated with OpenAI search discovery.
• ChatGPT-User: certain visits initiated by a person using ChatGPT.
• GPTBot: automatic crawling associated with model development.

Add a narrow robots.txt policy

A balanced policy can allow search discovery while declining GPTBot. Do not add a broad User-agent: * block unless you intend to affect ordinary search crawlers too.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Watch for: Robots.txt controls cooperating automatic crawlers. It is not authentication, and vendor documentation may describe user-requested fetchers differently.

Check the network layer

A permissive robots.txt file does not help if a CDN, WAF, hosting firewall, rate limit, or JavaScript challenge rejects the request. Review security logs for the documented user agent and verify traffic using vendor-published methods when available.

Avoid allowlisting a user-agent string alone. Anyone can copy one. Prefer documented IP verification or reverse-DNS procedures where the vendor provides them.

• Return a normal 200 response for public pages.
• Do not require a login, cookie challenge, or CAPTCHA for ordinary reading.
• Serve the title, headings, primary copy, links, and next action in HTML.

Make useful pages discoverable

Crawler access is only the doorway. Publish an XML sitemap containing canonical, indexable URLs and link important pages through ordinary HTML navigation. A crawler cannot recommend a page that contains no specific answer worth retrieving.

Use descriptive titles and headings, keep factual claims visible, and link to first-party sources. These basics serve conventional search and AI-assisted discovery at the same time.

Verification checklist

1. Fetch /robots.txt from the production hostname and confirm the intended rules.
2. Check one important page without cookies or authentication.
3. Confirm the initial HTML contains the page’s main answer and links.
4. Review CDN/WAF logs after a documented crawler visit.
5. Rescan the site after changing robots, cache, or firewall settings.