Maintained reference

AI crawler directory

One AI company may use different identities for search, user-requested visits, and model development. This directory keeps those purposes separate and links each entry to first-party documentation.

Discovery

AI search

Automated crawlers that build or improve search experiences and source discovery.

On request

User-directed access

A fetch made because a person asked an AI product to visit or use a page.

Separate choice

Model development

Crawlers or control tokens related to training or improving foundation models.

A policy is not one switch

Allowing an AI search crawler does not automatically mean allowing a training crawler. User-requested fetchers can also behave differently from automatic crawlers. Read the documented behavior before changing robots.txt, and verify important traffic with published IP ranges or reverse DNS where the vendor provides them.

OpenAI

3 documented identities
GPTBot Model development

Automatic crawler for content that may be used to improve and train OpenAI foundation models.

Documented control: OpenAI documents GPTBot as an independent robots.txt control.

Official source
OAI-SearchBot AI search

Automatic crawler used to surface websites in ChatGPT search features.

Documented control: OpenAI documents OAI-SearchBot as an independent robots.txt control.

Official source
ChatGPT-User User-directed

A fetcher used for certain actions initiated by a ChatGPT or Custom GPT user; it is not an automatic web crawler.

Documented control: OpenAI says robots.txt rules may not apply because the request is user initiated.

Official source

Anthropic

3 documented identities
ClaudeBot Model development

Automatic crawler for content that could contribute to training Anthropic generative AI models.

Documented control: Anthropic documents that its bots honor robots.txt directives.

Official source
Claude-User User-directed

A fetcher used when a Claude user asks for content from a website.

Documented control: Anthropic documents that its bots honor robots.txt directives.

Official source
Claude-SearchBot AI search

Automatic crawler used to improve the relevance and accuracy of Claude search responses.

Documented control: Anthropic documents that its bots honor robots.txt directives.

Official source

Perplexity

2 documented identities
PerplexityBot AI search

Automatic crawler used to surface and link websites in Perplexity search results; Perplexity says it is not used for foundation-model training.

Documented control: Perplexity documents robots.txt controls and publishes IP ranges.

Official source
Perplexity-User User-directed

A fetcher used when a person asks Perplexity a question that requires visiting a page.

Documented control: Perplexity says this user-requested fetcher generally ignores robots.txt.

Official source

Google

1 documented identity
Google-Extended Model development

A robots.txt control token for use of Google-crawled content in Gemini model training and certain grounding features; it is not a separate HTTP crawler.

Documented control: Google documents Google-Extended as a robots.txt product token that does not affect Google Search inclusion or ranking.

Official source

Common Crawl

1 documented identity
CCBot Open web dataset

A crawler that contributes to Common Crawl's public web archive, which can be used by many independent downstream projects.

Documented control: Common Crawl documents that CCBot follows robots.txt and publishes verification information.

Official source

Apple

1 documented identity
Applebot-Extended Model development

A robots.txt control token that determines whether content crawled by Applebot may be used to train Apple's foundation models; it does not crawl pages itself.

Documented control: Apple documents Applebot-Extended as a separate content-use control that does not remove a page from Apple search features.

Official source

How to use this directory

  1. Choose your policy. Decide separately whether you want AI search discovery, user-requested access, and model-development use.
  2. Inspect the real file. Check the robots.txt served by the production hostname, including rules inherited from broad wildcard groups.
  3. Check the network layer. A CDN or firewall can block a documented bot even when robots.txt allows it.
  4. Recheck the source. Crawler names, IP ranges, and behavior change; use the linked vendor page as the final authority.

Registry last reviewed: 2026-07-30.

This directory describes documented technical controls. It does not guarantee that a service will crawl, index, cite, train on, or exclude any particular page.

Check your policy

See what your website currently tells these systems.

Scan my website