Tools: Fix Your Robots.txt Or Your Site Disappears From Google 2026

Your site will be removed from Google search results if you don't have a robots.txt file or the Googlebot site crawler can't access it.

Here's the video from Google Support that covers it:

Adam Coster ran into a weird issue with site traffic and posted about it in the Shop Talk Show discord. Traffic incoming from Google looked like this:

The issues seemed to be that Google wouldn't index the site without a robots.txt file.

I can't imagine Google voluntarily slurping up less content. I went to see what I could find. Sure enough, I found this page from Google Support from July 23, 2025:

Fix 'robots.txt unreachable' Error ~ Website Not Indexing?

Your robots.txt file is the very first thing Googlebot looks for. If it can not reach this file, it will stop and won't crawl the rest of your site. Meaning your pages will remain invisible (on Google).

I haven't looked to see if this was a recent change, but it has to be. There's no way something so fundamental has just slipped by without becoming common knowledge.

But, the timeline doesn't matter. It's how things are now.

This absolutely blows my mind. I don't have tracking on my site. I never would have noticed this if someone hadn't pointed it out.

Source: HackerNews