Miroir

Miroir is a web-scraper/spider allowing you to backup ('mirror') websites for offline use. This could prove useful if you want to backup a blog or keep a copy of a website that may be shut down. A relatively large variety of engine options are made available, such as:

• Spider options: alter the ways robots.txt files are treated, parse JAVA files, accept cookies, and attempt to parse links even in javascript or other tags.
• Limit options: set the maximum mirroring depth (both internal and external), the maximum size of non-HTML or HTML files that are to be mirrored, the maximum size of the mirror, the maximum time spent mirroring, the maximum transfer rate, the maximum connections/second, and the maximum number of links.
• Download options: set the maximum number of connections, the timeout length, the number of retries (when the app fails to download a file), and the minimum transfer rate.
• Identity options: use a proxy, set up the default referrer URL, and set up HTTP headers.
• Build options: create log files, use a cache, alter the structure of the resulting copy, and make an index (websites can be downloaded 'as is' or can also be encapsulated in document files).
• Chosen mirroring options can be saved as presets.
• Several websites can be downloaded at the same time, if you tell the app to use a cache it continues where it left off.