maxwell

A fully automated, multithreaded, link-following, file-retrieving webspider.

More than just a program that reads HTML and retrieves files

Teleport Pro - the most fully-featured and robust webspider available.

Scan up to 40 million addresses in a single project