Using a robots.txt file

Welcome to Geeklog Monday, October 25 2021 @ 02:54 pm EDT

Using a robots.txt file | 8 comments | Create New Account

The following comments are owned by whomever posted them. This site is not responsible for what they say.

Using a robots.txt file

Authored by: JohnVanVliet on Monday, October 04 2004 @ 02:32 am EDT

yes it is BIG aprox 4 gig . it is for maps of the solar sys. for Celestia ( a 3d space sim.)

Using a robots.txt file

Authored by: Dirk on Monday, October 04 2004 @ 04:23 pm EDT

I had a similar problem with Googlebot last year where it downloaded one particular file over a 1000 times for no apparent reason. Fortunately, it was a small file and it did abide the robots.txt that I set up ...

bye, Dirk

Geeklog

Search

Resources

About

Getting started

Support

Development

Topics

User Functions

What's New

Articles last 4 weeks

Comments last 4 weeks

Pages last 4 weeks

Links last 4 weeks

Downloads last 4 weeks