Trakis

Trakis · Rules

Internal links to pages blocked by robots.txt

Internal links pointing at addresses that robots.txt disallows. Usually harmless: account, cart and login pages are meant to be blocked. It matters when a real content page is among them — it can never be read, however many links it gets.

NoticeCRAWL-006 · Crawlability · checked once for the whole site

Where it usually comes from

robots.txt rules that block private areas such as account, cart or checkout, while the menu or footer still links to them. That part is expected; the risk is a rule written too broadly.

How to fix it

  • Look at the blocked addresses in the finding.
  • If they are private pages (account, cart, login, settings), accept the finding as intentional.
  • If a content page is among them, remove the robots.txt rule that blocks it.

How to check it yourself

Read each Disallow line in robots.txt and make sure it matches only private areas. A rule like Disallow: /p also blocks /products.

What it looks like in a report

Notice 2 internally linked URLs are disallowed by robots.txt. · evidence: /account, /cart

Both are private pages, so this finding can be accepted as intentional and will stop appearing. (An example shop; the numbers are illustrative.)

How the audit counts it

CRAWL-006 is a notice in the crawlability category, and weighs lightly in that category’s score. It is a fact about the site as a whole, so it is reported once, not once per page.

If the finding is deliberate on your site, it can be accepted as intentional in the report. It then leaves the list and the score, and comes back on its own if what it shows changes.

Other crawlability checks

Find it on your site

A Trakis audit runs this check and the other 65 on your site, and tells you on how many pages each one sits. Or go back to all the rules.