Skip to content
Open dashboard

Help centerCrawl

Crawlability

Whether AI crawlers may read your site under robots.txt, compared with your competitors.

Open Crawlability

AI assistants can only use pages their crawlers are allowed to read. Crawlability reads your robots.txt for each known AI bot and checks whether those bots actually get through. It also compares your rules with your competitors’.

Three kinds of AI bot

Search
Builds the index an assistant searches when it answers, for example OAI-SearchBot or PerplexityBot. Blocking these keeps you out of answers.
Training
Collects content to train future models, for example GPTBot or Google-Extended. Many sites block these by choice, and that doesn’t affect today’s answers.
User requests
Fetches a page live because a person asked about it, for example ChatGPT-User. Some vendors say these may not follow robots.txt.

Bot access

One row per bot, grouped by the rule that applies to it: rules that name the bot, the global User-agent: * rules, or no rules at all. Filter by purpose, platform or status, or search for a bot.

Status

  • OpenNo paths disallowed
  • PartialAllowed, but some paths disallowed
  • BlockedThe homepage is disallowed
  • Unknownrobots.txt couldn't be read
Status describes what robots.txt declares for the whole site. Click a status to open robots.txt with the lines that restrict that bot highlighted.

For your own site, Accessshows whether a request sent with the bot’s name succeeded, compared with a normal request:

Access

  • Verified: the site answered the bot normally
  • Failed: the bot got an error a normal visitor didn't
  • Unknown: e.g. bot protection challenged it, or a normal request failed too
  • Not checked (competitor sites, rule-only tokens)

Hover an icon for the reason. Click a bot’s name to see its restricted paths, the rule for the homepage, and a link to the vendor’s documentation.

Checking again

Editors can click Reload robots.txt after changing the file. View robots.txt shows the file as it was read, and History lists what changed between checks.

Compare

One row per bot and one column per site, yours marked You. Competitor robots.txt files are read automatically and refreshed daily. Choose Differences onlyto see just the bots that sites treat differently. Click a cell to read that site’s robots.txt.

URL tester

  1. Paste a URL

    A page on your domain or a tracked competitor’s.

  2. Choose the check

    Leave Policy only to read the rules for every bot. Or pick Request as … to also request your own page as one bot.

  3. Click Check URL

    Each bot shows allowed or blocked for that page, with the exact rule that matched.

Fixing a blocked bot

  1. Find the highlighted lines in robots.txt (click the bot’s status).
  2. Remove the Disallow for search and user-request bots, or add an Allow for the pages you want in answers.
  3. Publish the file, then click Reload robots.txt and confirm the status is Open.

Recommendations suggests this fix automatically when a search bot is blocked, with the steps to follow. Training bots never trigger a suggestion; blocking them is your choice.

Related guides