Foundations
Indexation & Crawl Budget
Getting the right pages indexed and the wrong ones out, with robots, sitemaps and parameter handling that agree with each other.
What usually goes wrong
robots.txt, the meta robots tag, canonical tags and the sitemap are four different ways to say what should be indexed, and on most sites at least two of them disagree. The disagreement is silent.
You would notice
- Pages you blocked are indexed anyway
- The sitemap lists URLs that redirect or 404
- Filter and sort URLs appear in search results
- Search Console shows far more discovered URLs than you have pages
The work
Indexation & Crawl Budget, line by line.
Each line is work that gets done and can be checked afterwards. None of it is a promise about where you will rank — nobody can honestly make that one.
- 01What is indexed compared against what should be
- 02robots.txt, sitemaps and canonicals made to agree
- 03Parameter and faceted URLs brought under control
- 04Soft 404s and near-duplicates found and dealt with
- 05XML sitemaps rebuilt to list only indexable URLs
In practice
Four signals that have to agree
robots.txt Disallow: /search/ # crawler never fetches it
meta robots noindex, follow # ...so this is never seen
canonical https://example.com/shop/ # points elsewhere
sitemap.xml lists /search/?q=shoes # and invites it back in
Result: the page stays indexed, with no description,
and nobody can work out why.Blocking a URL in robots.txt stops the crawl, which also stops the noindex tag from ever being read.
Questions
Asked often enough to answer here.
What is crawl budget, in practice?
How much of your site a crawler bothers to fetch in a given period. It matters on large sites and is mostly irrelevant below a few thousand URLs.
Should I block filters in robots.txt?
Usually not. Blocking stops the crawl but not the indexing. Handling it with canonicals or noindex on a crawlable page works better.
My sitemap has 40,000 URLs and 8,000 pages. Is that bad?
Yes, and it is common. A sitemap should list URLs that are indexable, canonical and 200. Everything else in there is noise.
Also in Foundations
Work that usually travels with it.
- Technical SEO AuditA full crawl of the site against how search engines actually read it: status codes, canonicals, indexability, duplicate paths, and what is quietly blocking the rest.
- Site Architecture & Internal LinkingHow pages are grouped, how deep they sit, and how link equity moves between them. Most sites lose more here than they do on backlinks.
- Core Web Vitals & SpeedLCP, INP and CLS measured on real page loads rather than a lab score, then the render path fixed where the numbers say it hurts.
- Structured Data & SchemaValid JSON-LD for the page types that earn rich results, and removal of the markup that only earns warnings.
Contact
Want indexation & crawl budget on your site?
We will look at it and come back with what we would do and what it would cost. If we are not the right fit for it, we will say that instead.