Skip to content
SEO Tools
Help and documentation
SEO Critical impact Medium 10–20 min

Robots (indexing and robots.txt)

Controls what search engines may crawl and index.

What it is

Indexing is controlled by two things: the robots.txt file (what may be crawled) and the meta robots / X-Robots-Tag header (whether to index a page, e.g. noindex). They are easily confused.

Why it matters

Misconfiguration is one of the most destructive SEO mistakes: an accidentally left-over "noindex" or "Disallow: /" can drop the entire site out of Google. Conversely, missing rules let useless pages (cart, filters) be indexed and waste crawl budget.

How to do it

  1. 1Verify that production pages do NOT have noindex (a common leftover from development/staging).
  2. 2In robots.txt, don't block CSS/JS or pages you want indexed.
  3. 3Block only what shouldn't be in the index (admin, cart, internal search).
  4. 4Reference the sitemap in robots.txt.
  5. 5Remember: robots.txt prevents crawling but not necessarily indexing. To remove from the index, use meta noindex.

robots.txt

Wrong
User-agent: *
Disallow: /        ← blocks the ENTIRE site
Correct
User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /

Sitemap: https://www.example.cz/sitemap.xml

Conclusion

Indexing is the "master switch" of visibility, which is why it belongs among the critical checks. After every deployment, verify that no noindex remains from the staging environment and that robots.txt doesn't block anything important. One mistake here can cost all your organic traffic.

Related topics