Run This 10-Minute Robots.txt and Sitemap Check Before Google Misses Your Best Pages

Check your robots.txt and XML sitemap right now — misconfigurations here silently kill your rankings.

I've audited hundreds of sites. At least a third of them have something wrong with their robots.txt file, their XML sitemap, or both. The scary part? The site owners had no idea. Traffic just quietly bled out while Google ignored pages it should have been crawling — or wasted crawl budget on pages that didn't matter.

This is foundational technical SEO. Your robots.txt tells search engines what they're allowed to crawl. Your XML sitemap tells them what you want them to crawl. When these two files contradict each other — or when either one is broken — you've got a problem that no amount of content or link building will fix.

Here's the 10-minute check I run on every new client site. Do this today.

Step 1: Audit Your Robots.txt (3 Minutes)

1. Pull it up. Go to yourdomain.com/robots.txt in your browser. If you get a 404, you don't have one. That's not catastrophic — Google will crawl everything by default — but you're missing an opportunity to control crawl budget.
2. Look for overly broad Disallow rules. The most common mistake I see: Disallow: / blocking the entire site. This happens more than you'd think, usually left over from a staging environment that went live. If you see this, fix it immediately.
3. Check for blocked important directories. Look for Disallow lines that accidentally block your blog, product pages, category pages, or any section you actually want indexed. Common culprits: Disallow: /blog/, Disallow: /products/, Disallow: /*? (which blocks all parameterized URLs, including some you might need).
4. Confirm your sitemap is referenced. At the bottom of your robots.txt, you should see a line like: Sitemap: https://yourdomain.com/sitemap.xml. This helps search engines discover your sitemap automatically. If it's missing, add it.
5. Test it in Google. Open Google Search Console, go to Settings → Crawling → robots.txt, and check for any errors or warnings Google has flagged.

Step 2: Audit Your XML Sitemap (4 Minutes)

1. Find it. Try yourdomain.com/sitemap.xml or yourdomain.com/sitemap_index.xml. If you can't find it, check your robots.txt for the Sitemap directive. Still nothing? You need to create one. Yoast, Rank Math, or a standalone plugin can generate one automatically.
2. Open it and spot-check. Your sitemap should include every important, indexable page on your site. Scan for obvious problems: Are your top pages listed? Are there URLs returning 404s or redirects? A sitemap should only contain live, canonical, 200-status URLs.
3. Check the size. XML sitemaps max out at 50,000 URLs or 50MB uncompressed. If your site is large, you should have a sitemap index file pointing to multiple smaller sitemaps. Most CMS plugins handle this automatically, but verify it.
4. Look for junk. Tag pages, thin archive pages, paginated URLs, internal search results — these shouldn't be in your sitemap. You're telling Google “these are my important pages,” so only include pages that actually deserve to rank.
5. Submit it in GSC. Go to Indexing → Sitemaps in Search Console. Submit your sitemap URL if it's not already there. Check the status — it should say “Success.” If it shows errors, click through and fix them.

Step 3: Cross-Check for Conflicts (3 Minutes)

1. Never list a URL in your sitemap that's blocked by robots.txt. This sends Google mixed signals. Pick one: either you want the page crawled or you don't.
2. Never list a URL in your sitemap that has a noindex tag. Same logic. Your sitemap says “crawl this,” and noindex says “don't index this.” Clean it up.
3. Verify with a crawl. Run a quick crawl with Screaming Frog (free up to 500 URLs). Filter for pages that are in the sitemap but blocked, redirected, or noindexed. Fix every conflict you find.

Watch Out For This

If your site was recently migrated, redesigned, or moved from staging to production, check robots.txt first. I've seen entire sites deindexed because a developer left Disallow: / in the robots.txt after launch. It takes Google days to recrawl, and it can take weeks to recover your rankings. Set a calendar reminder to check this after every deployment.

This check takes 10 minutes and can surface problems that have been quietly destroying your organic traffic for months. Do it today, then revisit it quarterly.