TheArchivist
Your rating: Now say why...

(1) 1

Web spider script.   Free
Add to my Watch List
Email me when discounted
This script acts as a Web crawler, or spider. Starting with a particular URL it retrieves the Web page, scans it for links, and then attempts to retrieve all files linked to the page. This behavior repeats for each file retrieved and continues until one of several stop criteria is reached.

Stopping tests:
  • Number of hops (links) from the start page.
  • Links go "up" rather than "down".
  • Links go to disallowed servers.


If desired, the application will rewrite absolute URLs relative to the download hierarchy, producing a completely self-sufficient
What's New
Version 2.1: Corrected parsing of javascript function references, added php and asp to recognized "html" file extensions, fixed bug that truncated crawls done without the "legal servers" restriction.
Requirements
PPC, Mac OS 9 or later









  • SiteSucker
    +1
TheArchivist User Discussion (Write a Review)
ver. 2.x:
(1)
Your rating: Now say why...
Overall:
(1)

sort: smiles | time
burypromote


Anonymous reviewed on 10 Mar 2005
why is this an applescript?? there are many real applications, free, that do the same thing.
[Version 2.1]

1 Reply

burypromote
Anonymous commented on 25 Mar 2005
It was a good way to do it at the time (OS-9) and it offered a fine way to learn Applescript Studio. BTW, if you have a problem with it's function, you can write to me.
burypromote


Anonymous reviewed on 09 Mar 2005
does not work
[Version 2.1]


There are currently no troubleshooting comments. If you are experiencing a problem with this app, please post a comment.

There are currently no ratings. Write a comment or review now.

Downloads:2,438
Version Downloads:1,866
Type:Internet : Internet Utilities
License:Free
Date:09 Mar 2005
Platform:PPC 32 / OS X / OS Classic
Price:Free0.00
Overall (Version 2.x):
Features:
Ease of Use:
Value:
Stability:
Displaying 1-2 of 2
-
-
-
Please login or create a new
MacUpdate Member account
to use this feature
Watch Lists are available to
MacUpdate Desktop Members
Upgrade Now
Install with MacUpdate Desktop.
Save time moving files & cleaning
up space wasting archives.
This script acts as a Web crawler, or spider. Starting with a particular URL it retrieves the Web page, scans it for links, and then attempts to retrieve all files linked to the page. This behavior repeats for each file retrieved and continues until one of several stop criteria is reached.

Stopping tests:
  • Number of hops (links) from the start page.
  • Links go "up" rather than "down".
  • Links go to disallowed servers.


If desired, the application will rewrite absolute URLs relative to the download hierarchy, producing a completely self-sufficient archive.

This software is provided as "postcardware". You may download and use the software without cost so long as you register your use. You may register by email to this address.

[Please Note: At this time files referred to by plug-ins (e.g., Flash), Java and some Javascripts may not be identified. You should review the archive carefully for completeness when TheArchivist is done.]


- -