Can Google and AI bots read your site?
Check your robots.txt, sitemap and noindex tags — and see which search engines and AI assistants are allowed in.
Your result
–
Search & AI bots can they read your site?
Your robots.txt
Want to be found on Google and in AI answers?
We fix crawling and indexing problems, set up sitemaps, and get your site ready for AI search — then help it rank.
Good to know
Robots.txt & AI crawlers, explained
What is robots.txt?
A small text file at yoursite.com/robots.txt that tells search engines and other bots which parts of your site they may crawl. It also points them to your sitemap.
Should I block AI crawlers?
There are two kinds. AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch pages to show and cite them in answers — blocking them means AI assistants cannot recommend you. AI training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) collect content to train models — blocking those is a business choice and does not affect Google Search.
Does robots.txt remove a page from Google?
No. It only stops crawling. To keep a page out of Google, allow crawling and add a "noindex" robots meta tag.
Why is my sitemap important?
It lists every page you want found, so Google discovers new and updated pages faster. Link it from robots.txt and submit it in Google Search Console.
More free tools: see all free tools.