16-Oct-2004

Site slow

Apologies if the web site seems slow at the moment. I'm being hit by a couple of rogue spiders that ignore robots.txt and ignore the 403 status code.

In the last two days, before I banned the IP addresses, one of these bots had downloaded in excess of 100 MB. Even after the ban, it continues to hammer away, ignoring about ten to twenty 403 errors each second.

As it's coming from a university, I expect it's some student doing an assignment. I have sent an e-mail to their help desk and abuse addresses. We'll see if that stops the insanity...

UPDATE: Exceptionally quick response from the university's help desk allowed me to remove the bans from their site. Thanks guys. Now just one more bot to go...

Posted at October 16, 2004 10:59 AM
Tag Set:

Comments are closed