Welcome to Geeklog, Anonymous Thursday, September 28 2023 @ 07:42 pm EDT

Comments for Using a robots.txt file

Using a robots.txt file | 8 comments | Create New Account

The following comments are owned by whomever posted them. This site is not responsible for what they say.

Using a robots.txt file

Authored by: Dirk on Monday, October 04 2004 @ 04:23 pm EDT

I had a similar problem with Googlebot last year where it downloaded one particular file over a 1000 times for no apparent reason. Fortunately, it was a small file and it did abide the robots.txt that I set up ...

bye, Dirk

Search

Resources

About

Getting started

Support

Development

Topics

User Functions

What's New

Articles last 4 weeks

Comments last 4 weeks

Pages last 4 weeks

Links last 4 weeks

Downloads last 4 weeks