WebScraper

WebScraper uses the Integrity v6 engine to quickly scan a website, and can output the data (currently) as CSV or JSON.

• Easy to scan a site - just enter the starting URL and press "Go"
• Easy to export - choose the columns you want
• Plenty of extraction options, including HTML elements with certain classes or IDs, regular expressions, or entire content in a number of formats (html, plain text, markdown)
• Configuration of various limits on the crawl and the output file size

• Fixes problem causing scan to continue with the previous scan rather than starting a new one, if the previous one has been paused and then a new project opened or started
• Fixes a problem with scanning locally (file://) - scan was extending beyond the local files if there were external links on the pages
• Fixes helpers not working with local HTML files
• Fixes problem with class helper, if the class /id list was too long and needed scrolling, the 'hover to highlight' functionality wasn't working beyond the area originally visible
• Live view also greys-out external links - they won't be scraped, but they are in the live view table because they're part of the crawling process
• Adds field "information page" alongside blacklist and whitelist fields; this is useful if your information appears on detail pages which can be identified with a partial URL (/mip/ in the case of yellowpages.com), and you want to collect information from such pages, but not parse the links on those pages