Is your website blocking ChatGPT?

Open yourdomain.com/robots.txt in a browser. If you see Disallow: / under GPTBot, OAI-SearchBot, PerplexityBot, Bingbot or under User-agent: *, some AI tools are being told to stay out. Remove those lines for the crawlers that power search answers, and check your host's bot settings too.

Why this matters for a local contractor

When a homeowner asks ChatGPT "who's a good emergency plumber near me?", the answer is built from pages its search crawler has been allowed to read. The same goes for Perplexity, Microsoft Copilot and Google's AI Overviews. A site that turns those crawlers away gives them nothing to quote, so they quote your competitors.

This happens by accident more often than you'd think. In our October 2026 audits of 20 home service websites in the Tulsa area, 9 refused requests from at least one AI crawler, usually through a hosting or security setting the owner had never seen. One site's robots.txt blocked Googlebot and Bingbot as well, which hides it from regular search too.

Which crawlers to know

Each AI company runs more than one crawler. Some collect training data, and some fetch pages for live answers. You can treat them differently.

For a business that wants customers, the ones that matter most are the search crawlers: OAI-SearchBot, ChatGPT-User, PerplexityBot, Bingbot and Googlebot. Blocking the training crawlers is a business choice, and it won't hurt your chances in search answers.

How to check your site in two minutes

1. Type your web address followed by /robots.txt into a browser, for example yourbusiness.com/robots.txt.
2. Look for a User-agent line naming any crawler in the table above, followed by Disallow: /. That means "stay out of the whole site."
3. Check the User-agent: * group too. A Disallow: / there blocks every crawler that isn't named elsewhere.
4. If the file is empty, missing or only blocks admin folders like /wp-admin/, robots.txt isn't your problem.

The block you can't see in robots.txt

Some hosts and security services block AI crawlers at the server, before robots.txt is ever read. Cloudflare, for example, began blocking known AI crawlers by default for new sites in July 2025, and it can also add its own rules to your robots.txt. If your site sits behind Cloudflare, open its security or bot settings and look for an option about AI crawlers. WordPress security plugins can do the same thing.

A free audit checks this from the outside: we request your pages the way each crawler does and report which ones get turned away.

A robots.txt you can copy

This version lets search and AI answer crawlers in, keeps them out of a WordPress admin area and points them to your sitemap. Replace the address with yours.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://www.yourbusiness.com/sitemap.xml

If you'd rather keep your content out of AI training, add a group for GPTBot with Disallow: /. Search answers will still be able to find you through the crawlers above.

Common questions