SEO glossary · Updated
Crawling in SEO: how search engines find your pages
Crawling is the process where search engine bots (crawlers or spiders) discover and download pages on the web, mostly by following links.
Founder, SEO·AEO·GEO
Builds SEO·AEO·GEO and writes the crawler, schema, and citation research published here.
Crawling is the first stage of search. Programs called crawlers, spiders or bots, such as Googlebot, fetch pages from the web: they start from known URLs, follow links to find new ones, and download the text, images and code they find. A page that's never crawled can't be indexed or ranked.
How crawlers find your pages
- Links from pages they already know, on your site or others.
- XML sitemaps you submit or list in robots.txt.
- Direct submissions, such as URL Inspection in Search Console or IndexNow for Bing and other engines.
What stops crawling
| Blocker | Effect |
|---|---|
| Disallow rules in robots.txt | The crawler won't fetch the URL |
| Firewalls, bot protection, rate limits | The crawler gets an error or challenge page instead of content |
| Server errors (5xx) and timeouts | Google slows its crawling down |
| Pages with no links to them (orphans) | Hard to discover |
| Links that need JavaScript or clicks | Crawlers may not follow them |
Crawl budget
Crawl budget is how many URLs Google can and wants to crawl on your site. Google says it mainly matters for very large sites (around a million pages or more) or sites with many pages that change daily. Small sites rarely need to think about it, as long as the server is fast and reliable.
AI crawlers
AI companies run their own crawlers: some for search and citations (OAI-SearchBot, Claude-SearchBot, PerplexityBot), some for training (GPTBot, ClaudeBot, Google-Extended as a control token). Each can be allowed or blocked separately in robots.txt. See the AI crawler list.
SEO spider tools
SEO spiders such as Screaming Frog SEO Spider crawl your site the way a search engine does, to find broken links, redirects and missing tags across many pages. See SEO audit tools compared.
Check it on your own site
What the free audit tests for this
- robots.txt rules
- Edge access
- Crawl delay
- sitemap.xml
- Internal links
One URL, about 20 seconds, no login. You get the result of every check and the exact fix for the top three issues.
Audit your website freeFAQ
What's the difference between crawling and indexing?+
Crawling is fetching the page; indexing is analyzing and storing it so it can appear in results. A page can be crawled but not indexed.
How often does Google crawl my site?+
It varies by site and page: popular, frequently updated pages are crawled more often. Search Console's Crawl stats report shows your site's activity.
Can I make Google crawl faster?+
Not directly. A fast, reliable server, good internal links and an up-to-date sitemap help. Google doesn't accept payment to crawl more often.
Sources
- In-depth guide to how Google Search works — Google Search Central (2025-12-18)
- Introduction to robots.txt — Google Search Central
- Crawl budget management for large sites — Google Search Central
- Build and submit a sitemap — Google Search Central