Connection failed. Please try again.

Skip to content
SEO Tools

Robots.txt Checker

Enter your website URL and immediately find out whether your robots.txt is correctly configured and isn't blocking search engines from crawling your most important pages.

Share

What the tool checks

  • Existence and accessibility of the robots.txt file
  • Allow / Disallow rules and which sections are blocked
  • Access permissions for major crawlers (Googlebot, Bingbot, and others)
  • XML sitemap reference inside robots.txt
  • Common mistakes — accidentally blocking the entire site or CSS/JS files

Why it matters

Robots.txt is the first file a search engine fetches when it visits your site. A single bad rule — "Disallow: /" — can wipe your entire site from the index overnight. Missing a sitemap reference slows down the discovery of new pages. Getting robots.txt right protects your crawl budget and keeps bots focused on the pages that matter.

Detailed guide in the help center

Frequently asked questions

What is robots.txt?

A plain-text file placed in the root of your website (e.g. example.com/robots.txt) that tells search engine crawlers which parts of your site they are allowed or not allowed to visit.

Can robots.txt get my site de-indexed by Google?

Yes. A "Disallow: /" rule blocks crawling of your entire site. Pages that can't be crawled will typically drop out of the index or stop being updated.

Does robots.txt need to include a sitemap?

It's not required, but strongly recommended. Adding "Sitemap: https://example.com/sitemap.xml" helps search engines discover all your pages faster.

Rate this tool

Related tools

XML Sitemap Checker & ValidatorRedirect Checker (301 / 302)Structured Data Tester (Schema.org / JSON-LD)

Want a complete SEO + GEO website analysis?

Run all 20 checks at once, track rankings on Google and Seznam, and get improvement suggestions.

Run full analysis