Spider / Web Crawler

TL;DR

Web crawlers, known as spiders or bots as well, crawl across the World Wide Web to index pages for search engines, so the results given after searching a specific keyword are relevant.

What is a web crawler? 

A web crawler, spider, or search engine bot (such as Googlebot or Bingbot), crawls, downloads and indexes content from all over the Internet by automatically accessing a website and going through all the links within it.

How do web crawlers work?

Search engines almost always operate bots in order to collect, index, and provide relevant links in response to user search queries (keywords).

The role of a search engine crawler bot is very similar to a person organizing an entire library and creating an easy to check page or catalog of all the books to find them easily; similarly, the bots organize the pages across the web and display the relevant ones based on what a person searches.

New pages and sites are updated and published every second, and the web crawlers' main goal is to start crawling from a list of known URLs, and continue as they will find hyperlinks to other URLs, and they add those to the list of pages to crawl next.