How Google's Site Crawlers Index Your Site - Google Search
Description
Before you search, site crawlers gather information from across hundreds of billions of webpages. Discover what software Google uses to crawl and index your site so that they can be displayed in Google search results.