Do we have to have a robot.txt (Technics)

by Auge, Thursday, February 05, 2009, 20:07 (5551 days ago) @ crossitover

Hello

(like the content of my database which I want it to be indexed).

The database will be not indexed. Any robot will only index the content in your site. This could be the forum (and the sites of the threads) too.

However the thing is actually I don't even have a robot.txt file in any directories of my forum. Could this be a problem causing google not indexing my forum? My understanding is if I do not use robot.txt then I'm welcoming google to visit/index everything I want in my site. But it looks like google did not find robot.txt then it left!!!

You can have robots.txt in every directory but it should be enough to have it in the root directory of the website (where you'll find your homepage (maybe index.*, default.* ...)).

I don't know the actually behaviour of googlebot. But I know that googlebot uses the robots.txt and follow its instructions. Note: In the robots.txt you can only forbid the crawling of directories and/or files.

Tschö, Auge


Complete thread:

 RSS Feed of thread