Google has just pushed some updates to the reduce the Google crawl rate documentation. Most of the changes are just organizational to the content on the page but under the “Urgently reduce crawler traffic (for emergencies)” section, Google added new information on how to use Retry-After HTTP header.
Google said it has “Restructured the emergency crawl rate reduction section and added information and examples about the Retry-After HTTP header to the Reduce the Google crawl rate documentation.” Google added it did this to “Support for the Retry-After HTTP header is not new (it was already documented in Temporarily pause or disable a website), and adding it directly to the crawl rate reduction guide makes it easier to find and understand when urgently reducing crawler traffic.”
If you compare the live doc to the archived doc, you will mostly see a lot of things moved around the page.
The biggest addition is the section on:
When returning a 503 or 429 status code, you can also include a Retry-After HTTP header (as defined in RFC 9110 HTTP Semantics) to indicate when Google’s crawlers can retry the request, using either a delay in seconds or an absolute UTC date and time:
Google added example code as well, here is a screenshot:
Google isn’t introducing a new way to throttle Googlebot, the underlying mechanism hasn’t changed. Google already used 500, 503, and 429 responses as signals to reduce crawling. The new documentation makes Retry-After an explicit additional signal that site owners can provide when using 503 or 429.
Forum discussion at X.
