Crawl budget optimization and crawl architecture for large sites
Maximizes Googlebot crawl efficiency by prioritizing important pages and blocking crawl budget waste.
Ensure Googlebot spends its crawl budget on the most important site pages, accelerating indexing of new content and avoiding unnecessary crawling of low-value pages.
At a glance
Access
Free prompt
Open to copy — no account or payment needed.
Prompt objective
Ensure Googlebot spends its crawl budget on the most important site pages, accelerating indexing of new content and avoiding unnecessary crawling of low-value pages.
Real use case
ModaExpress, a fashion marketplace with 85,000 URLs, noticed in server logs that Googlebot crawls 3,000 pages/day but 60% are filter pages, pagination, and internal search parameters. Meanwhile, 15,000 new product pages added over the past 3 months still haven't been indexed.
Customize these fields first
Replace the placeholders with your own context before you run the prompt. That usually improves the first output more than adding more instructions later.
Prompt
Optimize crawl budget for the site [SITE URL], which has [NUMBER] pages and receives [NUMBER] crawls/day from Googlebot. **Phase 1 — Server log analysis:** Using log analysis data (Screaming Frog Log Analyzer or similar): - Total Googlebot requests in the last 30 days - Distribution by page type: product, category, blog, filter, pagination, internal search, parameters - Most crawled pages vs. most important pages (cross-reference with organic traffic) - Status codes returned: 200, 301, 302, 404, 500 - Crawl rate: requests/day and time patterns - Identify crawl traps: infinite URLs, duplicate parameters, calendars **Phase 2 — Waste diagnosis:** - URLs crawled that should NOT be crawled (% of total) - Important URLs that are NOT being crawled (list them) - Average indexing time for new content (publish to index) - Orphan pages: pages in sitemap that don't receive crawl **Phase 3 — Optimization plan:** **robots.txt:** - Block: filter URLs with parameters, internal search, duplicate tag pages, admin areas - Allow: main categories, products, blog, landing pages - Crawl-delay (if needed) **Meta robots and canonical:** - Pages with noindex,follow: [list types] - Pages with canonical to main version: [list patterns] - Pagination: rel=canonical for page 1 vs. self-referencing canonicals **Optimized sitemap.xml:** - Remove noindex URLs from sitemap - Segment sitemaps by type: products.xml, categories.xml, blog.xml - Include correct lastmod (actual modification date) - Prioritize new URLs with index sitemap **Internal linking for crawlability:** - Ensure important pages are ≤3 clicks from home - Breadcrumbs with Schema markup - Optimized mega menu (don't list all subcategories) **Phase 4 — Implementation and monitoring:** - Test changes on staging before production - Monitor Google Search Console: Pages indexed, crawl stats - Resubmit updated sitemaps - Verify in 2 weeks: new page indexing rate - Goal: reduce crawl waste by [%] and accelerate indexing to <[NUMBER] days Deliver the optimized robots.txt and sitemap structure.
Open directly in an AI — the text is pre-filled:
How to use this prompt
- 1Replace the key placeholders first: SITE URL, NUMBER, list types, list patterns.
- 2Replace any bracketed placeholders like [this] with your own context.
- 3Add extra background information when you want more tailored results.
- 4Combine multiple prompts in one conversation when you need a richer output.
- 5Save your best-performing prompts so they are easy to reuse later.
Next best step
Open the guide first, then branch only if you still need more.
A practical guide for marketers who want campaigns, prompts, and automations that shorten execution time.
If this prompt is close but not quite right, generate variants next. If the job is recurring, move into the course library after the guide.
Related prompts
View allAI Search Visibility Audit for Course Pages
Reviews whether a course, guide, or landing page is clear enough for search engines and AI assistants to cite.
Best for
Improve page clarity, citation readiness, and conversion paths for AI search, organic search, and human buyers.
Core Web Vitals audit and performance optimization roadmap
Diagnoses Core Web Vitals issues (LCP, INP, CLS) and creates a prioritized technical remediation plan.
Best for
Improve the site's Core Web Vitals metrics to meet Google's 'good' thresholds, ensuring performance is not a limiting factor for ranking.
Complete Schema Markup implementation for rich results
Plans and implements structured data (Schema.org) to win rich results and increase CTR in search results.
Best for
Implement Schema markup on all relevant site pages to enable rich results in Google (stars, FAQ, breadcrumbs, products, how-to), increasing visibility and organic CTR.
Canonical Tag Strategy and Duplicate Content Management
Audit and fix duplicate content issues using canonical tags, hreflang, and URL parameters.
Best for
Resolve all duplicate content issues on the site through proper canonical tag implementation, avoiding authority dilution and Googlebot confusion.
Explore other prompt categories
Move sideways into adjacent libraries when the current category is not the full answer.
Every prompt here is free. The course teaches the thinking behind them.
Copy as many prompts as you like. When you want to move from single prompts to a repeatable AI workflow, Learn AI in 30 Days walks through it, one day at a time.
Buy the course once ($15/$20 by length), or go all-access for $10/mo with a verifiable certificate.