Fundamentals

Web Crawling: What It Is, How It Works, and Why It Is Not Indexing

By Daniel Sardá · Published on October 21, 2026

6 min read1,185 words

Web crawling lets automated programs discover and visit pages. Understanding its stages and limits prevents confusion with indexing, ranking, or surveillance.

Web crawling is the automated process through which a program discovers web addresses and requests their resources for examination. That program is commonly called a crawler, bot, or web spider. For a search engine, crawling makes it possible to find pages that it can later process and consider for its index.

Here, “crawling” does not mean following a person’s activity. Nor does it, by itself, mean copying data from a site, adding a page to a search engine, or deciding where it will appear in results. The term is used for several distinct activities, and separating them helps explain what happens when an automated system moves through the web.

Key idea: Crawling means discovering and fetching resources; indexing means deciding whether their content belongs in an index. They are related stages, not synonyms.

How a crawler discovers pages

A crawler needs a starting point. It may begin with one or more URLs it already knows because it visited them before, because other pages link to them, or because they appear in a list such as a sitemap. These starting addresses are often called seed URLs.

When it requests a page, the program receives its content and may identify links to other URLs. Those new URLs can enter a list of possible visits. The cycle then repeats: request a resource, examine it, discover links, and choose what to visit next.

Imagine that a site publishes a new page and links to it from its homepage. When a crawler returns to the homepage, it may find that link and add the page to its crawl path. Discovery does not guarantee an immediate visit, however. The program may prioritize some URLs, defer others, or not crawl them at all, depending on its own rules, capacity, and objectives.

The so-called crawl frontier is a useful model for the list of known addresses that are still awaiting evaluation or a visit. Not every system is built in the same way, but the model makes clear that a crawler does not move at random or necessarily follow every link it finds.

Google explains that its search engine discovers pages through, among other channels, links and sitemaps, and that not every discovered URL is crawled. This describes how Google operates; other crawlers may use different priorities and procedures.

Crawling, indexing, and ranking are separate decisions

Once a search engine fetches a page, it must still interpret it and decide whether to add it to its index. Indexing is the process of analyzing and storing information about a resource so it can be retrieved later. A crawled page may remain out of the index for technical, quality, duplication, or other policy-related reasons.

Even being indexed does not ensure that a page will appear for every query. It certainly does not guarantee a particular position. Ranking takes place when the search engine selects and orders results in response to a specific search. It depends on a later assessment and is not an automatic reward for allowing crawling.

The sequence can be summarized as follows:

1. The system knows or discovers a URL.
2. It decides whether to crawl it and requests the resource.
3. It processes the content and decides whether to index it.
4. When a query is made, it evaluates whether to show it and in what order.

Each step has its own conditions. That is why saying that a page “has already been crawled” provides only limited information about its effective presence in search results.

Useful distinction: Crawling enables a search engine to learn about content, but it is not a promise of indexing, visibility, or rank.

It is not the same as scraping or user tracking

Crawling and scraping can both involve programs making automated requests, but they emphasize different purposes. Crawling seeks to traverse resources and discover addresses. Scraping focuses on extracting specified data: prices, text, catalogs, or other elements. One system may perform both tasks, but the concepts are not interchangeable.

It also differs from user tracking, which records people’s actions or characteristics: visits, clicks, identifiers, or journeys across services. A crawler traverses web resources. A tracking system observes activity associated with users or devices. The fact that both processes may leave technical records on servers does not make them the same practice.

This distinction matters because the normative questions differ as well. Crawling raises issues such as a site’s published access rules and the cost of serving automated requests. Tracking people brings privacy, consent, and the use of personal data into view.

What `robots.txt` and `noindex` do

Site operators have signals they can use to communicate how they want certain agents to access or handle their pages. Two of the best known serve different functions.

The `robots.txt` file publishes rules for identified crawlers and can ask them not to access particular paths. The Robots Exclusion Protocol standard makes clear that these rules are not an authorization mechanism or a security measure. A bot may disregard them, and the paths written in the file are public.

Blocking a path in `robots.txt` therefore does not make its content private. If an area contains restricted information, it must be protected with authentication and real access controls. An administrative page behind a sign-in is protected by an access barrier; a path mentioned only in `robots.txt` is not.

The `noindex` rule, by contrast, tells compatible search engines not to include a resource in their index. It can be declared in a page’s meta tag or in an HTTP header. There is an important condition: the search engine needs to access the resource to read the instruction. Google’s documentation on `noindex` notes that if its crawler cannot enter because of a `robots.txt` rule, it will not see that instruction.

Practical caution: `robots.txt` manages crawl requests from cooperative bots; `noindex` communicates an indexing decision; access control protects private content.

The limits of web crawling

The web is not a perfectly connected catalog. Some pages have no visible links, some resources require credentials, some addresses are generated dynamically, and servers may limit requests. Crawlers also operate with finite time, computing capacity, and bandwidth. Discovering a URL does not require a visit, and visiting a page once does not imply returning at a particular frequency.

These constraints also explain why the choices of a site operator and those of a crawler operator should remain distinct. Publishing content for open access allows different agents to try to request it, but it does not erase the operator’s legitimate interest in setting rules, protecting resources, and managing infrastructure costs. At the same time, a preference expressed to bots does not substitute for a technical barrier when access truly needs to be restricted.

Understanding web crawling requires keeping these conceptual boundaries intact. It is the mechanism that lets many tools explore resources and discover pages, not a guarantee of inclusion in search engines or a universal license for every later use. Knowing which stage is at issue—discovery, access, extraction, indexing, or result selection—makes it possible to choose the right control and assess each problem more precisely.