How to Keep Cloudflare From Blocking Google and AI Crawlers

AI Visibility

September 8, 2026

•

Corey Lydstone

A dealership website can have strong content, accurate inventory and clean structured data and still be hard for search and AI systems to use if the security layer in front of the site blocks their crawlers.

That risk is getting more important for websites running through Cloudflare. In August, Search Engine Journal reported an anecdotal case in which a site owner found Googlebot and Bingbot receiving 403 errors while Cloudflare's AI training bot block was enabled. Turning the block off restored access. The report did not establish that Cloudflare itself was malfunctioning, but it highlighted a larger issue Cloudflare has already documented: starting September 15, 2026, mixed-purpose crawlers can be affected by AI training blocks.

September 15 is the date to watchCloudflare says crawlers used for both search and AI training will be included in its Training policy beginning September 15. As of this writing, Cloudflare classifies Googlebot as a mixed-purpose crawler that is also used for AI training and will block it by default starting September 15th. The same applies to the other mixed-purpose crawlers unless the site's policy allows them.

For car dealers, this is not an abstract bot-management issue. Google says a page must be accessible to Googlebot to be eligible for indexing, and its AI Overviews and AI Mode rely on the same core Search index. OpenAI likewise tells publishers to allow OAI-SearchBot if they want site content to be eligible for summaries and snippets in ChatGPT search.

The goal is not to allow every automated request on the internet. It is to make sure Cloudflare does not accidentally stop the legitimate search, AI search and user-directed crawlers that can help shoppers find your dealership. If your website provider manages Cloudflare or has its own security policy, make these changes only after confirming the configuration with that provider.

Why Cloudflare's settings are easy to miss

Cloudflare now gives site owners several ways to control crawler traffic. The controls do not all live in the same place. Some appear on the domain overview, others live in AI Crawl Control, and still others sit inside Security settings.

That means checking a single "AI bots" toggle is not enough. A dealership can allow a crawler in one place and still block it somewhere else.

Example configuration for maximum crawler accessIf a dealership and its website provider want the broadest possible access for legitimate search and AI crawlers, the configuration below shows one way to set up Cloudflare. Search and Agent access are the most directly related to current discoverability. Allowing Training as well avoids mixed-purpose crawler conflicts, but dealerships should confirm that choice with the team responsible for the website before making changes.

Where to check	Recommended setting
Domain overview → Manage AI bot access	Do not block (allow crawlers)
Domain overview → Manage your robots.txt	Disable robots.txt configuration for broadest crawler access
AI Crawl Control → Security	Block Crawler switches off for crawlers you want to allow
Security → Settings → Configure AI bot policies	Search: Allow Agent: Allow Training: Allow
Security → Settings → Block AI bots	Set to: Mixed purpose crawlers will continue to be allowed

The Cloudflare settings to check

1. Allow AI bot access for the domain

Start from the Cloudflare domain list. In the right-hand column, find Manage AI bot access.

Set it to:

Do not block (allow crawlers)

On the domain overview, set Manage AI bot access to “Do not block (allow crawlers).”

This prevents Cloudflare's broad AI-bot control from deploying a managed rule that blocks the crawlers it classifies as AI training crawlers.

2. Stop Cloudflare from adding its managed AI rules to robots.txt

In the same right-hand area of the domain overview, find Manage your robots.txt.

For the broadest crawler access, set it to:

Disable robots.txt configuration

Disabling this setting stops Cloudflare from managing the site's robots.txt file.

This setting needs one important explanation. Disabling Cloudflare's robots.txt configuration does not remove your site's own robots.txt file. It only stops Cloudflare from adding and maintaining its own managed crawler directives.

Cloudflare's current managed robots.txt configuration includes search=yes and ai-train=no, along with explicit Disallow: / rules for several AI training crawlers such as GPTBot, ClaudeBot, Google-Extended and Bytespider. That can be a reasonable policy for publishers that want search access while opting out of training. It is not the broadest possible crawler policy.

If your goal is AI search visibility but you do not want AI trainingyou do not have to allow every training crawler. Search and training are different. This guide uses the broadest allow configuration because it is the simplest way to avoid Cloudflare's mixed-purpose crawler conflict. Sites with legal or content-licensing restrictions should choose a more selective policy.

After disabling Cloudflare's management, open your site's own file at:

https://yourdealerwebsite.com/robots.txt

Make sure the site's CMS, website provider or another system is not blocking Googlebot, OAI-SearchBot or any other crawler you intend to allow.

3. Check individual crawlers in AI Crawl Control

Next, use Cloudflare's main navigation and open:

AI Crawl Control → Security

Cloudflare shows the crawlers it has detected on the site, along with request activity and a Block Crawler switch.

Cloudflare AI Crawl Control crawler list showing Googlebot, ChatGPT-User, BingBot and other crawlers with Block Crawler switches off

Review individual crawler access. In this screen, the Block Crawler switches are off, which means those crawlers are allowed.

For crawlers you want to reach the dealership website, make sure the Block Crawler switch is turned off.

At minimum, pay close attention to search crawlers such as Googlebot and Bingbot, along with AI search or user-directed crawlers that can surface or retrieve your dealership information.

Cloudflare's own AI Crawl Control documentation recommends allowing crawlers that provide value through citations, referrals or other useful services. It also notes that AI Crawl Control can enforce a block even when a crawler is otherwise able to request the site.

4. Set Cloudflare's new AI bot policies to Allow

Now open:

Security → Settings

Scroll to the Bot traffic section and select Configure AI bot policies.

The newer AI policy controls and the legacy Block AI bots setting both appear in the Bot traffic section.

Cloudflare's newer policy system separates crawler behavior into three categories:

Category	What it does
Search	Crawls or indexes content so it can appear in search results.
Agent	Pulls information from a site to answer a user's question or perform a user-directed task.
Training	Collects content to train or fine-tune AI models. Cloudflare's Training category can also include mixed-purpose crawlers used for Search.

For maximum crawler access, set all three to:

Allow (do not block)

For the broadest discoverability configuration, set Search, Agent and Training to “Allow (do not block).”

Search and Agent are the most directly connected to current discoverability. A dealer does not have to allow training-only crawlers to appear in ChatGPT or other AI search products. OpenAI, for example, separates OAI-SearchBot, which supports ChatGPT search discovery, from GPTBot, which is associated with model development.

The reason Training deserves extra care in Cloudflare is that some crawlers serve more than one purpose. Cloudflare's interface warns that blocking Training can block crawlers that use the same bot for Search.

5. Before September 15, allow mixed-purpose crawlers

Still inside Security → Settings → Bot traffic, open:

Block AI bots [Deprecating on September 15]

Set the legacy scope to:

Do not block (allow crawlers)

Then, in the September 15 notice, select:

Mixed purpose crawlers will continue to be allowed.

This is the time-sensitive setting. Select “Mixed purpose crawlers will continue to be allowed” before Cloudflare retires the legacy control on September 15.

Why we recommend dealers allow Training too

Search and Agent access are most directly connected to whether AI systems can find and use dealership information today. Training is different, but we still believe most dealers should allow it.

For publishers, content may be the product. They depend on pageviews, ads or subscriptions, so letting AI systems learn from their content can create a real downside.

Dealers have different incentives. Their website exists to help sell and service vehicles. The goal is not to protect the pageview. It is to make sure shoppers and the AI systems assisting them understand the dealership, its inventory, services and reasons to buy there.

Allowing training crawlers does not guarantee that an AI model will learn or recommend a dealership. But we see little strategic advantage for most dealers in blocking legitimate AI systems from learning from their public information.

That is why our default recommendation for dealerships is to allow Search, Agent and Training crawlers, while continuing to block malicious or unwanted bots.

The legacy Block AI bots panel is itself being deprecated on September 15, so someone reading this article after that date may no longer see the exact screen above. The newer Configure AI bot policies controls are the long-term settings to check.

Why this matters for Google and AI search

Google's technical requirements are straightforward: Googlebot cannot be blocked, the page must return a successful response, and the page needs indexable content. Google also tells site owners that appearing in AI Overviews and AI Mode depends on the same basic Search eligibility, including allowing crawling through robots.txt, hosting infrastructure and content delivery networks.

That makes Cloudflare part of the AEO and SEO stack whether a dealership thinks of it that way or not. A CDN or security rule that returns a 403 to Googlebot sits upstream of everything on the page. Better page copy, schema and internal linking cannot compensate for a crawler that never reaches the content.

The same principle applies to AI search. OpenAI says OAI-SearchBot should be allowed if a publisher wants content to be discoverable and eligible for summaries and snippets in ChatGPT search. The crawler also needs to be able to pass through the site's host or content delivery network.

Do not stop with the five settings

These settings cover the most obvious Cloudflare controls, but a site can still have conflicting security rules elsewhere.

After making the changes, check these too:

Your own robots.txt file: confirm the site's CMS or provider is not blocking crawlers you want.
WAF and custom security rules: look for broad bot blocks, managed challenges or country rules that could catch legitimate crawlers.
Super Bot Fight Mode: if your plan uses it, verify that legitimate verified bots are not being blocked.
Cloudflare crawler activity: review unsuccessful requests and 403 responses in AI Crawl Control and Security Analytics.

Cloudflare notes that AI Crawl Control uses its Web Application Firewall to enforce crawler actions. In practice, that means crawler access should be tested as an end-to-end result, not assumed from one dashboard setting.

Verify that the site is actually crawlable

After updating Cloudflare, test important dealership pages rather than assuming the configuration worked.

Open Google Search Console.
Use URL Inspection on the dealership homepage and several high-value pages.
Run Test Live URL.
Confirm Google can fetch the page and is not being blocked by robots.txt or server access restrictions.
Review Cloudflare AI Crawl Control for unsuccessful crawler requests.

Useful pages to test include the homepage, new and used inventory, an individual vehicle page, service pages, location and hours, and high-value model or local content.

It is worth checking again after September 15, especially if the site previously used Cloudflare's legacy AI bot block.

The practical takeaway for dealers

Cloudflare is doing an increasingly sophisticated job of separating helpful automation from unwanted scraping. The tradeoff is that site owners now have more crawler controls to understand.

For a dealership trying to maximize search and AI visibility, a broad-access configuration typically allows Search and Agent crawlers, makes an explicit choice about Training, and keeps mixed-purpose crawlers such as Googlebot and Bingbot from being caught in a broader AI-training block.

The most immediate action is to open Cloudflare before September 15 and confirm that “Mixed purpose crawlers will continue to be allowed” is selected. Then walk through the other settings above and test the site from the crawler's point of view.

Check with your website provider before changing these settingsMany dealership websites are managed by a website provider, agency, IT team or other vendor that may also control Cloudflare, hosting, security rules or bot settings. Before changing anything in Cloudflare, confirm with the team responsible for your website that the recommended settings are appropriate for your specific setup. A change that is right for one dealer site can conflict with another site's security, hosting or provider configuration. This article is general technical guidance, not a substitute for instructions from the team that manages your website.

See what AI says about your dealership

CarGenius Dealer AI Visibility Reports measure which dealerships AI assistants actually name when shoppers ask non-branded Find, Buy, and Service questions, which competitors win those recommendations, and which sources are shaping the answers. See where your store stands.

View the Visibility Reports Get a Custom Report Or book a demo →

Sources

More from AI Strategy

AI Visibility

September 8, 2026

•

Corey Lydstone