Moving your website to a CDN: keep Googlebot crawling

Google's Search Central team posted on 11 Nov 2026 about sites that move behind a CDN such as Cloudflare. Two things can go wrong in the first days: your own server gets more crawler traffic, not less, and a blanket bot rule can stop Googlebot. If you are planning a CDN move, set the bot rules first, warm the cache, then watch Search Console.

Why a new CDN can mean more load at first

Google says its crawlers adjust their speed to how fast a site answers. Once your domain points to a CDN, pages come back much quicker, so Google may decide the site can take more requests and slowly crawl faster.

The CDN cache starts empty, though. The first request for each page still has to go back to your own server (Google calls it the origin) before the CDN keeps a copy. Visitors mostly open the same popular pages, which get cached quickly. Googlebot works through every page on the site, so for a while many of its requests still land on your server.

How a bot rule can drop you from Google

When the graphs jump, it is tempting to switch on a "block all bots" or challenge setting in the CDN dashboard. Google warns that Googlebot cannot pass a CAPTCHA. If it keeps hitting a challenge or a 403 page, Google treats the site as blocked or broken, crawls less, and your pages, including shopping listings, can start to fall out of Search.

What to do

1. Before the switch, open the bot and firewall settings in your CDN. Turn on the option that lets verified bots through (Google says most major CDNs and firewalls have one) instead of a rule that challenges every automated visitor.
2. If you need to stop a bad bot, target that bot by name or with robots.txt. Leave the rest alone.
3. Check that a visitor calling itself Googlebot is real before you block or allow it. Google's method: a reverse DNS lookup on the IP should end in googlebot.com, google.com or googleusercontent.com, and a forward lookup of that name should give back the same IP. Google also publishes its crawler IP ranges as JSON files.
4. Warm the cache. Before you send all traffic through the CDN, open your busiest pages, any upcoming promotion pages and the pages Google crawls most, a few at a time, so the CDN already holds copies.
5. If your server still struggles, Google allows a short brake: return 429 or 503 to crawlers for a few hours, at most a day or two. Do not leave it longer.
6. After the move, open Search Console, go to Settings, then Crawl stats. Watch the number of requests, the server response codes and host availability. A rise in 403, 429 or 5xx answers is the first sign something is blocking Google. Google says the report matters most for sites with more than about a thousand pages. On a smaller site, we suggest running URL Inspection on a few key pages instead.

If your origin server is small, a CDN will not fix that during the first days. A hosting plan with room to spare, such as our enhanced performance hosting, gives your site more headroom while the cache warms up.

Source and credit

We are sharing this news from Google Search Central. This summary is written by Synctech in our own words.

• Moving to a CDN without overwhelming your origin (or blocking Googlebot), Google Search Central Blog, 11 Nov 2026: https://developers.google.com/search/blog/2026/11/crawling-cdn
• Verify requests from Google crawlers and fetchers, Google for Developers: https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
• Crawl Stats report, Search Console Help: https://support.google.com/webmasters/answer/9679690 Retrieved 12 Nov 2026. Google, Googlebot, Search Console and Cloudflare are trademarks of their owners. Synctech is not connected with them. This is general information, not legal, tax or financial advice. If anything here is wrong, email support@synctech.com.my and we will correct it.