Layered navigation: from millions of URLs to a manageable crawl
A category with ten filters of ten values each, three sort options, three view modes and four page limits theoretically produces more URLs than Google will ever crawl. In practice, log files show Googlebot spending a large share of its time on sort and limit parameters and filter combinations with no search demand at all, while new products wait days to be indexed.
We set a rule per parameter: sort and view parameters out of the crawl via robots, filter pages canonicalised to the category, and the filter combinations that do have demand (brand plus category is the classic) as their own indexed landing pages with content, through an extension or a dedicated category. Then we watch the logs to confirm the crawl pattern shifts towards the pages that matter. More about crawl budget and index management.