I’m an idiot.* Google continues not to index my relatively new site, Project Mai Tai.* Possible reason?* I forgot to create a robots.txt file for it!* D’oh!
I’ve heard Google will bounce a site for not having that.* If that’s what’s been keeping me out, I’m happy to say the Google bots come along frequently, so with any luck they’ll index the site soon.
In the meantime, I feel like an idiot.*
Technorati Tags: Project Mai TaiProject Mai Tai

Robots.txt file - Read More...

You're not an idiot but you heard it wrong.

Googlebot doesn't care whether there's a robots.txt or not.

This is clearly a myth.

I concur that it's a myth. The purpose of a robots.txt file is to restrict spider access, not grant it. So the absense of a file just lets the spiders run unrestricted. Additionally, the methods to index a site could just be defined in the meta tags for that site: the ol' "index, follow."

I've operated a number of sites that didn't use robots.txt files and had no issue getting in Google SERPs (Search Engine Results Pages).

File a robots.txt file under one of those steps you undertake in the creation of a site "just in case" there is a search engine out there that relies on it. To state that Google requires robots.txt files is spreading inaccurate information.

I agree too, it's a myth.
The bots look for it though and you'll see that in your errors log.
No big deal.