This script acts as a Web crawler, or spider. Starting with a particular URL it retrieves the Web page, scans it for links, and then attempts to retrieve all files linked to the page. This behavior repeats for each file retrieved and continues until one of several stop criteria is reached.
- Number of hops (links) from the start page.
- Links go "up" rather than "down".
- Links go to disallowed servers.
If desired, the application will rewrite absolute URLs relative to the download hierarchy, producing a completely self-sufficient archive.
This software is provided as more...