Trakis

Trakis · Rules

Sitemap URLs blocked by robots.txt

Addresses listed in sitemap.xml that robots.txt forbids crawlers to fetch. The sitemap asks search engines to crawl a page and robots.txt forbids it in the same breath. The page is not read, and the contradiction makes the sitemap less trusted as a whole.

CriticalCRAWL-001 · Crawlability · checked once for the whole site

Where it usually comes from

A page was blocked in robots.txt during development or a redesign, while the sitemap plugin kept listing it. Sometimes a whole folder is disallowed and one of its pages later became important.

How to fix it

  • Decide, for each listed address, whether it should be in search at all.
  • If it should: remove the Disallow line in robots.txt that covers it.
  • If it should not: take it out of the sitemap. A sitemap should list only pages you want indexed.

How to check it yourself

Open yoursite.com/robots.txt and yoursite.com/sitemap.xml side by side, or test a listed address with the robots.txt tester in Google Search Console.

What it looks like in a report

Critical The sitemap lists /collections/sale, and robots.txt disallows /collections/.

The sitemap and robots.txt disagree about one folder; fixing either file settles it for every page inside. (An example shop; the numbers are illustrative.)

How the audit counts it

CRAWL-001 is a critical in the crawlability category, and weighs heavily in that category’s score. It is a fact about the site as a whole, so it is reported once, not once per page.

If the finding is deliberate on your site, it can be accepted as intentional in the report. It then leaves the list and the score, and comes back on its own if what it shows changes.

Other crawlability checks

Find it on your site

A Trakis audit runs this check and the other 65 on your site, and tells you on how many pages each one sits. Or go back to all the rules.