WebScraper

WebScraper uses the Integrity v6 engine to quickly scan a website, and can output the data (currently) as CSV or JSON.

• Easy to scan a site - just enter the starting URL and press "Go"
• Easy to export - choose the columns you want
• Plenty of extraction options, including HTML elements with certain classes or IDs, regular expressions, or entire content in a number of formats (html, plain text, markdown)
• Configuration of various limits on the crawl and the output file size

• Adds option to open a list of urls to be scraped in a local text file (or csv with a single column containing the urls)
• Adds File > Open List of Links for this purpose
• Updates the engine to inherit many improvements and fixes from the Integrity crawler

• Fixes a problem causing regex extraction and helper to fail if page uses ISO-8859-1 encoding

Requirements