Googlebot
TL;DR
Googlebot is Google's crawler used to automatically scan websites by following links from one page to another and adding them to the index based on relevance.
What is Googlebot?
Googlebot, as the name says, is Google's crawler (also named bots) in charge of crawling websites. Googlebot is the general name given to:
- A desktop crawler that simulates a user on the desktop (Googlebot Desktop).
- A mobile crawler that simulates a user on a mobile device (Googlebot Smartphone).
How does Googlebot access your site?
The purpose of Googlebot is to crawl and scan all web sites and their pages via links: basically, Googlebot finds and reads all the new or updated content and adds it to the index based on its relevance.
The visits created by the Google crawlers may be displayed in the referrer logs, and they should be specified in robots.txt.
Googlebot crawls as many pages from a site as possible but respecting the disavow links selected by the website owner as well.
To ask Google to (re)index a website, please note these general guidelines:
- Crawling can take anywhere from a few days to a few weeks, but you can always monitor progress using either Google’s tools: Index Status report or the URL Inspection tool.
- Requesting a recrawl multiple times for the same URL or sitemap won't get it crawled any faster.
- The best way to verify that a request actually comes from Googlebot is to use a reverse DNS lookup on the source IP of the request.
- Google actively fights spammers; if you notice spam pages or sites in Google Search results, you can report spam to Google.
- If you want to prevent Googlebot from crawling content on your site, you have a number of options. Be aware of the difference between preventing Googlebot from crawling a page, preventing Googlebot from indexing a page, and preventing a page from being accessible at all by both crawlers or users.
