Blocked by robots.txt

The Page indexing report lists the page as URL blocked by robots.txt; URL Inspection says Blocked by robots.txt. Both mean the same thing: a rule in the site’s robots.txt applies to Googlebot and forbids this address. Google’s help page for it has three steps: confirm the block, find the rule, change or remove it.

It stops crawling, not indexing. Google’s robots.txt guide says robots.txt is not a way to keep a page out of Google: a blocked address that is linked from elsewhere can still appear in results, without a description.

• In Search Console, inspect the URL. Under Page indexing, “Blocked by robots.txt” confirms it.
• Open https://the-same-host/robots.txt in a browser: the file crawlers get, on the exact host of the blocked URL. www.example.com and example.com each have their own, and some hosts and CDNs add lines your own file doesn’t have (Cloudflare does).
• Paste it into the robots.txt validator with the blocked path. The Googlebot row says allowed or blocked, and shows the exact line that decides. It runs in your browser; nothing is sent anywhere.
• Search Console’s robots.txt report, under Settings, shows which version Google fetched and when, with the versions of the last 30 days.

Most surprises come from one of these, all from Google’s robots.txt specification:

• Googlebot obeys one group. If the file has a User-agent: Googlebot group, Googlebot follows that group and ignores User-agent: * entirely. A group added “just for Googlebot” with one line can quietly drop every other rule.
• Paths are prefixes. Disallow: /blog also blocks /blog-tips and /blogging. Write /blog/ for the folder only, or /blog$ for that one address.
• Paths are case-sensitive. Disallow: /Admin does not block /admin.
• The longest match wins, and when an Allow and a Disallow are equally long, Google uses the less restrictive one.
• The file is cached for up to 24 hours, so a fix can take a day to count.

A narrow exception, written the way Google reads it:

• The sitemap lists pages robots.txt forbids. You ask Google to crawl a page and forbid it in the same breath. Trakis lists these (CRAWL-001).
• Your own links point at blocked pages. That is one of the ways Google finds the addresses in this report. Trakis lists the links (CRAWL-006).
• A whole folder was blocked for a reason that no longer holds: a development area, an old section, a filter that later became the main category page.
• A platform writes the file for you. Google’s help notes that hosting services handle robots.txt in different ways; look for the setting in yours before editing a file it will overwrite.

• Remove or narrow the line that matches the page, or add an Allow for the one path that should be open.
• Check it again in the validator, with the live file.
• In the robots.txt report, choose Request a recrawl for the file. Google says it is rarely needed, but useful right after unblocking.
• Inspect the most important pages, run the live test, and request indexing. Then use Validate fix in the Page indexing report if it is offered.

The opposite surprise: the page is in Google even though robots.txt blocks it, found through links. If you want it in search, unblock it so Google can read it. If you want it out, unblock it too and add noindex: Google’s noindex guide says a page blocked by robots.txt is never read, so its noindex is never seen.

Is “blocked by robots.txt” an error? Only for pages you want in search. For pages you meant to block it is the file working.

Does blocking GPTBot or other AI bots block Googlebot? No. Each crawler reads its own group; only rules under User-agent: * or Googlebot apply to Google Search. Blocking Google-Extended doesn’t affect Search either.

Should I block CSS and JavaScript files? Not the ones a page needs. Google’s guide says not to block resources whose absence makes the page harder to understand.

I fixed robots.txt. Why is the page still listed? The file can be cached for a day, and each page is rechecked when Google crawls it again. Request a recrawl of the file and inspect the page; the count goes down over days.

What if robots.txt answers with an error? A 404 means no rules: Google crawls everything. A server error is worse: Google stops crawling the site for up to 12 hours, then works from the last copy it had.

The Trakis audit reads robots.txt together with every page it crawls: blocked pages in the sitemap, links into blocked folders, and which search engines and AI assistants may come in. It counts each problem once, with the pages it covers. Create a free account and add your site, or see every Search Console status, explained.