Indexed, though blocked by robots.txt
Google has a page in its index that it has never read. Robots.txt told it not to fetch the page, links pointed to it anyway, and Google indexed the address from the links. The fix turns on one question: do you want this page in Google or not?
What Google is telling you
The page is counted as indexed, with a warning. Google’s help page puts it plainly: Google “always respects robots.txt, but this doesn’t necessarily prevent indexing” when someone links to the page. It won’t fetch the page, but it can still index the address.
What shows in results is that address, with no description, Google’s robots.txt guide says, because Google never saw what the page says. The same guide is blunt about the cause: robots.txt is not a way to keep a page out of Google.
One question decides the fix
| You want the page… | What to do |
|---|---|
| In Google | Remove the robots.txt line that blocks it, so Google can read it. Find the line. |
| Out of Google, but still on the site | Remove the block and add noindex. Google can only obey a noindex it is allowed to read. |
| Gone for good | Delete it so it answers 404 or 410. |
| Private | Put it behind a login. |
| Out of results today | The Removals tool hides it for about six months, then do one of the above. |
Why you have to unblock a page to remove it
It sounds backwards. Google’s help for this status says “robots.txt is not the correct mechanism to avoid being indexed”, and its noindex guide explains why: a page blocked by robots.txt is never fetched, so its noindex is never seen, and links can keep the address in the index. Two mistakes follow from that: adding a Disallow line to remove a page, which is how this status starts, and adding noindex while leaving the Disallow in place, which does nothing.
Where these URLs come from
- Links into a blocked folder. From other sites, the case Google’s help describes, and from your own menus, filters and footers. Trakis lists your own links to blocked pages (CRAWL-006).
- A sitemap that lists blocked pages. It asks Google to come in while robots.txt keeps it out (CRAWL-001).
- Wildcard rules. A line such as Disallow: /*? blocks every address with parameters; the ones that are linked anywhere can end up here.
- A robots.txt you didn’t write. With no file on disk, WordPress serves its own, and plugins can add lines to it:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpIf the URLs in the report are not under /wp-admin/, the line that blocks them came from a plugin or from a robots.txt someone added. WordPress’s “Discourage search engines” setting does not touch robots.txt since version 5.3; it adds a robots meta tag instead.
Take pages out, step by step
- Group the URLs in the report by folder or pattern, and decide each group with the table above.
- For the groups that should leave Google, remove or narrow the Disallow line, and check the live file in the robots.txt validator.
- Add <meta name="robots" content="noindex"> to those pages, or an X-Robots-Tag: noindex header for PDFs and other files that have no HTML.
- Take them out of the sitemap.
- In the robots.txt report, request a recrawl of the file; then inspect a few of the URLs and run the live test, which should now report the noindex.
- Leave the block off. Put the Disallow back and Google stops reading the noindex, and the links can bring the addresses back.
As Google recrawls them, the pages move from this warning to “URL marked ‘noindex’” under not indexed, which is what you wanted.
Common questions
Is it an error? A warning. If you wanted the page in Google, it is there but unread; if you wanted it out, it isn’t.
Why does it name a page my robots.txt doesn’t block? Check the file on the exact host and protocol of the URL: http and https, www and the bare domain each have their own. Google may also be working from a copy up to a day old; request a recrawl of the file and inspect the URL again.
Can I just use the Removals tool? Only for a while. Google says a removal lasts about six months, after which the page can come back unless it is gone, behind a login or marked noindex.
How is this different from “Blocked by robots.txt”? There the blocked page is not indexed. Here it is. See Blocked by robots.txt.
Check the whole site
The Trakis audit reads robots.txt with every page it crawls and lists blocked pages in your sitemap and your links into blocked folders, each counted once with the pages it covers. Create a free account and add your site, or see every Search Console status, explained.
Last changed 13 October 2026.