Updates on Google: Documentation for Googlebot Verification
With regard to visits from IPs connected to GoogleUserContent.com, Google updated the official Googlebot documentation.
Google updated their Search Central documentation to verify Googlebot. Adding documentation about user-triggered bot visits is information that was missing from previous Googlebot documentation and has created confusion for many years, with some publishers blocking the IP ranges of the legitimate visits.
Newly Updated Bot Documentation
Google added new documentation that categorizes the three different kinds of bots that publishers should expect.
Google bots are into three types:
- Googlebot search crawler.
- Special-case crawlers
- Users can activate fetchers (Google User Content).
Due to Google’s lack of explicit documentation, that last one, GoogleUserContent, has long baffled publishers.
According to Google, GoogleUserContent is as follows:
Users-activated fetchers
Tools and product features that let the user initiate a fetch.
As an illustration, Google Site Verifier responds to user requests.
These fetchers disregard robots.txt restrictions because the fetch was put forward by a user.
According to the documentation, the following domain will appear in the reverse DNS mask: –***.gae.googleusercontent.com
Recent updates by Google to their Google Crawlers page include a section on user-triggered fetchers.
The same bots contain each crawler’s list, but the page has been rearranged to separate user-triggered fetchers into their own group.
As of right now, the following crawlers are user-triggered fetchers:
- Google Podcasts, Google News, and PubSubHubbub all use Feedfetcher to crawl their RSS or Atom feeds.
- Google News landing pages, the Google Publisher Center retrieves and processes feeds that publishers voluntarily provide through the Google Publisher Center.
- Google Read Aloud: text-to-speech (TTS) technology by Google Read Aloud to retrieve and read web pages at the user’s request.
- Google Site Verifier: Upon user request, the Google Site Verifier retrieves the verification tokens for Search Console.
Before it disappears from Google’s search engine results pages (SERPs). The SEO community believed that when a user viewed a website through the translate button in the search results, there was bot activity coming from IP addresses connected to GoogleUserContent.com.
However, what we now know about user-triggered fetchers is what was above.
Google also added the following details regarding user-triggered fetchers:
Users-activated fetchers Users can activate user-triggered fetchers to carry out a task specific to a given product. Google Site Verifier, for instance, responds to user requests.
Because they are user-driven fetchers, they frequently disregard robots.txt instructions. The user-triggered-fetchers.json object contains information about the IP ranges that the user-triggered fetchers use.
According to Google’s updated documentation, the Google Site Verifier tool may detect bot activity coming from IP addresses connected to GoogleUserContent.com.
The documentation also now refers to googleusercontent.com when discussing IP addresses for the domain name GoogleUserContent.com.
Finally, Google stopped supporting its Android Mobile Apps crawler.
Both the full string and the user agent token were: AdsBot-Google-Mobile-Apps
We now know that bot activity from IPs connected to GoogleUserContent is not spam or hacker bots. In fact, they are from Google. Publishers who have blocked IP addresses connected to GoogleUserContent in the past should probably unblock them now.
