Fix in this order across all 25 sites: anything that blocks crawlers or indexing, then the discovery files (robots.txt, sitemaps, canonicals, llms.txt), then structured data. Within each tier, sort by how many important pages a fix touches and how long it takes. A blocked site gets fixed before any schema work starts, however small the block is to correct.
This article gives you a ladder, a simple scoring method, a worked triage table and a way to report the backlog to clients.
Why the order matters
Each tier depends on the one before it. Schema on a page that crawlers cannot fetch does nothing. Google's structured data guidelines say not to block pages carrying structured data with robots.txt, noindex or other access controls [1]. A perfect Organization block on a noindexed home page is wasted effort.
A sitemap that lists blocked or redirected URLs wastes effort for the same reason. The upstream fix has to come first.
Tier 0: anything that blocks access
Check these first, on every site, before scoring anything else.
Disallow: /underUser-agent: *, or under a specific AI crawler's user agentnoindexin a meta robots tag or anX-Robots-Tagheader on templates that should be indexed- The home page or key templates returning 4xx or 5xx to bots
- A firewall, bot-protection or login layer that serves bots a challenge page instead of content
- Redirect loops or long chains on the main entry URLs
Two things to keep straight. robots.txt controls crawling, not indexing. A URL blocked in robots.txt can still appear in results without a snippet, and noindex only works if the crawler is allowed to fetch the page and see it. A site can therefore have two separate problems that look alike.
For the AI-crawler side of this check, see how to check if AI can crawl your website. Whether to block a given crawler is a client decision. Your job in triage is to find blocks nobody chose on purpose.
Sweep all 25 sites in one pass
Put the domains in domains.txt, one per line, and run:
while read d; do
code=$(curl -s -o /dev/null -w "%{http_code}" "https://$d/")
robots=$(curl -s "https://$d/robots.txt")
blanket=$(echo "$robots" | tr -d '\r' | grep -ci '^disallow: /$')
hdr=$(curl -sI "https://$d/" | grep -ci 'x-robots-tag:.*noindex')
meta=$(curl -s "https://$d/" | grep -ci '<meta[^>]*name="robots"[^>]*noindex')
echo "$d home=$code root_disallow_lines=$blanket noindex_header=$hdr noindex_meta=$meta"
done < domains.txt
This is a first filter, not an audit. root_disallow_lines counts every Disallow: / line, including ones under user agents you did not intend. Open robots.txt for any site that scores above zero. A bot-protection layer will not show up here, because curl may be treated differently from a real crawler. Check those separately in server or WAF logs.
If you move on to bulk URL checks through the Search Console API, expect usage limits on its resources [2]. Batch and space your requests.
Tier 1: discovery files and signals
Once nothing is blocked, check whether crawlers can find the right URLs and tell which version is canonical.
| Item | What to look for |
|---|---|
| robots.txt | Exists, returns 200, references the sitemap, no stray rules |
| XML sitemap | Exists, lists only indexable 200 URLs, honest lastmod |
| Canonicals | Each key page points to its own preferred URL, matching the sitemap |
| URL normalization | One host and one slash rule, enforced with 301s |
| llms.txt | Present if the client wants it, consistent with robots.txt |
Google's documentation on consolidating duplicate URLs covers rel=canonical and the other methods available [3]. For the details, see the guides on XML sitemaps and llms.txt. llms.txt is a reading guide and does not replace the access checks in Tier 0.
Tier 2: structured data
Schema comes last because it only pays off on pages that are reachable and correctly indexed. Google recommends JSON-LD for eligible rich results [1]. Within this tier, a reasonable order is:
- Organization and WebSite on the home page, so the entity is named one way everywhere
- Article or equivalent on content templates
- Page-specific types (FAQ, breadcrumbs, product) where they match visible content
The schema markup guide has JSON-LD examples. Fix a template once and every page on that site benefits, so template-level schema scores better on effort than page-by-page edits.
Scoring within a tier
Tiers set the order between categories. For ranking items inside a tier, a simple scale is enough. This is a working convention for sorting a backlog, not a measured model.
- Impact (1–3): 3 = affects the home page or most key templates; 2 = affects a section; 1 = affects a few pages.
- Effort (1–3): 1 = a config or file edit; 2 = a template or plugin change; 3 = needs a developer, client sign-off or a migration.
Do high impact, low effort first. Flag anything with effort 3 early, because the wait for approval is usually longer than the work.
A worked triage table
These findings are hypothetical, written to show the method. They are not real client data.
| Site | Finding | Tier | Impact | Effort | Order |
|---|---|---|---|---|---|
| A | Disallow: / in robots.txt | 0 | 3 | 1 | 1 |
| B | noindex header on all blog URLs | 0 | 3 | 2 | 2 |
| C | WAF challenge page served to unknown bots | 0 | 3 | 3 | 3 |
| D | Sitemap lists 301 and noindex URLs | 1 | 2 | 1 | 4 |
| E | No sitemap reference in robots.txt | 1 | 2 | 1 | 5 |
| F | www and non-www both return 200 | 1 | 3 | 2 | 6 |
| G | No Organization schema on home page | 2 | 2 | 1 | 7 |
| H | Article schema missing on blog template | 2 | 2 | 2 | 8 |
Site C is Tier 0 but effort 3, so open the client conversation on day one and work through the quick wins while waiting. Tier 0 items from different sites all outrank every Tier 1 item, even though D and E are cheaper.
Reporting the backlog
Keep one table for the whole portfolio, with one row per finding:
| Column | Purpose |
|---|---|
| Site | Which client |
| Finding | One sentence, specific |
| Tier | 0, 1 or 2 |
| Impact / Effort | The 1–3 scores |
| Owner | You, client, or client's developer |
| Status | Found, Proposed, Approved, Fixed, Verified |
| Verified on | Date and how (curl output, header, fetched page) |
Three practices keep it honest:
- Mark "Fixed" and "Verified" separately. A change deployed but not re-checked is not done.
- Report in tiers, not in counts. "Two sites had blocked crawlers; both fixed and verified" is clearer than "34 issues resolved."
- Do not promise results. Report what changed and how you confirmed it. Whether AI engines or search pick up a fix depends on their own crawl schedules, and no public documentation gives a guaranteed timeline.
Re-run the sweep script on a schedule. Blocks often return after a redesign, a CDN change or a plugin update, and a monthly sweep catches them.
What to do next
- Put all 25 domains in a file and run the Tier 0 sweep today.
- Open robots.txt manually for any site the script flags, and check bot protection in your logs.
- Create the backlog table and enter every finding with its tier and scores.
- Fix and verify Tier 0, then Tier 1, then start on schema templates.
- For a per-site checklist to hand to a developer, use the technical SEO checklist.