KSF

From KallestadWiki

The Kallestad Spider Framework

The Kallestad Spider Framework is basically a highly configurable spidering utility written in Perl. I have had the need for spidering software on various occassions to perform various tasks. I wanted to build a repeatable solution that could do whatever I wanted with minimal configuration hassle. I also wanted something structured enough to the point where I didn't have to keep reinventing the wheel every time I wanted to spider a group of pages.

What you can do with Spiders