Posted on: 09/27/06 06:27am
By: Robin
Hello,
currently Geeklog provides this functionality with this file, robots.txt:
User-agent: *
Disallow: /comment.php
Disallow: /docs
Disallow: /pingback.php
Disallow: /submit.php
Disallow: /trackback.php
It means: All bots leave the following files and directories.
I would like to prevent bots from indexing the whole site. Is it enough and whether this works:
User-agent: *
Disallow: /
TIA
Robert
Prevent search engines from indexing
Posted on: 09/27/06 07:55am
By: jmucchiello
Yep. Assuming they obey the robots.txt file. Almost all search engines do but you can't really stop the ones that don't.
Prevent search engines from indexing
Posted on: 09/27/06 09:29am
By: Robin
[QUOTE BY= jmucchiello] Yep. Assuming they obey the robots.txt file. [/QUOTE]
Thanks Is there another way to achieve that?
Prevent search engines from indexing
Posted on: 09/27/06 02:48pm
By: Dirk
What exactly are you trying to accomplish?
The big search engines (Google, Yahoo, MSN, Ask.com, ...) follow the robots.txt standard. There are always a few
clueless bots[*1] by smaller companies, but those don't really matter anyway.
Unless you had security or spam prevention in mind, but that would be different topics.
bye, Dirk