SEO guides
Understand robots metadata and noindex
Robots metadata can tell a supported crawler that a page should not be indexed or that links should be treated in a particular way. A private or internal search page may deliberately use noindex. A public article may contain it accidentally. Intent matters more than a red or green indicator.
What to do
- Decide which pages should appear in search. Administration pages and internal search results often serve different purposes from public guides.
- Look for general robots and crawler-specific tags. Interpret conflicting values for the crawler they address.
- Also check server X-Robots-Tag headers if you have access. Local HTML analysis has not fetched HTTP headers.
Example
<!-- Only if exclusion is intentional -->
<meta name="robots" content="noindex, follow">Check for yourself
- Is noindex intentional here?
- Are tags or headers conflicting?
- Is confidential content protected by real access control?