Blocked due to other 4xx issue

Google’s help page calls it URL blocked due to other 4xx issue: the server returned a 4xx error not covered by any other status in the report, and Google suggests debugging the page with the URL Inspection tool. The 4xx codes that do have their own lines are 401, 403 and 404.

Whatever the code, Google’s page on HTTP status codes says it treats every 4xx answer but 429 the same way: it does not use the content, does not index the URL, and removes it from the index if it was there. These answers do not change how fast Google crawls the rest of the site.

• In URL Inspection, run Test live URL on one of the listed addresses. It fetches the page the way Google does, now, and shows whether the fetch succeeded.
• From a terminal, curl -s -o /dev/null -w "%{http_code}" https://example.com/the-page prints the code of an ordinary request, such as 400. (curl -I asks with HEAD, which some servers answer differently.)
• If the two disagree, the page opens for you and is refused to Google, something between them decides by who is asking: a firewall, bot protection, a rate limit, a rule that blocks a country or a range of addresses.
• In your server or CDN logs, look up Googlebot’s requests for the address: the code is there, and often the rule that answered.

429 Too Many Requests is the exception: Google says it treats it as a sign that the server is overloaded, as a server error, and slows its crawl. A rate limit that counts Google as one visitor too many is a server question: Server error (5xx). And a 403 has its own status and guide: Blocked due to access forbidden (403).

Only when the address is one you want in search. Filter the report to the URLs in your sitemap first: an address there that answers 4xx is the site contradicting itself. Then check where Google found the rest. An address your own pages link to is worth fixing at the link, or at the page; one nothing leads to any more, answering 410 because you removed it, can stay as it is.

The Trakis audit reports each case where its fix is:

• A page that answers 4xx and that your links lead to, with the pages the links are on (LINK-004).
• An address in your sitemap that answers 4xx (CRAWL-010).
• Any other page that does not answer 200, with its code (CRAWL-002).

• Answer 401 or 403 to slow Google down. Google’s status code page says not to use them to limit the crawl rate. It slows down for 429 and server errors.
• Switch off the protection to make the list go away. Find the rule that refuses Google and make an exception for verified search engine crawlers.
• Validate before you know the code. Fix the cause for the pages you want, then use Validate fix for those.

Why does it say “blocked”? Because the server refused the request. Nothing in robots.txt is involved; that is a different status, “URL blocked by robots.txt”.

The page opens fine for me. Why does Google get a 4xx? Something answers Google differently: a firewall, bot protection, a rate limit or a country rule. The live test in URL Inspection shows what Google gets now.

Will it hurt the rest of the site? Google says 4xx answers other than 429 do not change its crawl rate. The cost is the listed pages themselves, if you want them in search.

The Trakis audit reads every page and reports every answer that is not 200 where its fix is: the link that leads to it, the sitemap that lists it, or the page itself, with the code each one gave. Create a free account and add your site, or see every Search Console status, explained.