Support: Resources

Web Scraper Plus+ Manual

Here is a PDF version of the manual.

SQL Express Management Studio

Free Microsoft Tool which allows you to run queries on your local PROVIDUSSTD database. Download from Microsoft.

Next Pages Package Template

Copy this package to use as a starting poing for all of your packages! Contains recommended settings like script tag removal and one second bandwidth throttling between requests. Also contains counter datapage template (you should copy this as well and specify database.) Contains steps needed for a typical next pages scrape, with SQL queries included! Please see video tutorials for an in depth explanation of a next pages package.

Download: Default.ppk

Chunked package templates

Extracting over 10,000 rows? Copy these packages and refer to this forum post for more info.

Download: Setup Chunked.ppk
Download: Details Chunked.ppk

SQL Tutorial

Web Scraper is heavily-based in SQL. Here is a tutorial to get started with the basics of SQL.

HTTPLook

A useful program for sniffing out what forms are acutally submitting. See their website and download the trial version.

WebMon

A free utility which allows a user to monitor a webpage (the entire page or specific parts of it) at given intervals to see when it is changed. If it is, this can launch a package automatically. (Right-click and choose "Save Target As" to Download)

MoveFile.exe

A command line program that allows you to move all files in a directory (and sub-directories) to a single directory based upon size, filename, and extension. Useful for sorting files once you have crawled them. See a screenshot for its usage. (Right-click and choose "Save Target As" to Download)

ChangeIP.exe

Allows you to set your IP address, as well as change a number of other settings. See a screenshot for its usage. (Right-click and choose "Save Target As" to Download)

Zip Code Excel/CSV Sheets

Want to submit information to a form by zip codes? Here is a list of all major US zip codes, along with their respective city and state in Excel and CSV (Comma Separated Values file).

Brute Force 3 Character Dictionary

Allows you to submit all combinations of 3 characters to a form. Excel or CSV

Brute Force 2 Character Dictionary

Same concept, but with 2 characters. Excel or CSV

English Language Dictionary

Huge list of English words, and some combinations with numbers. CSV (too large a file for Excel)

Company Name Permutations

List of thousands of companies. CSV

Counter Table

Here is a web-based counter table that you can scrape any time you need a quick counter table. The table is useful, for example, for cycling through pages. Simply use this URL: http://www.velocityscape.com/Counter.aspx?start=1&end=500&step=1 (feel free to change the numbers) or fill out the following boxes with integers to get the table. And here is the datapage to scrape it.


Start
End
Step


Alternatively, you can create this table directly in SQL. The following SQL command simply creates an array of numbers between @start and @end increasing by the amount @step. Execute this command in Query Commander or Query Analyzer:


 DECLARE @start INTEGER, @end INTEGER, @step INTEGER

 SET @start = 1 -- change this to suit your needs

 SET @end = 500 -- also change

 SET @step = 1 -- also change

 CREATE TABLE _COUNTER (_COUNT VARCHAR(20))

 WHILE @start <= @end

 BEGIN

  INSERT INTO _COUNTER (_COUNT)

  VALUES (CAST(@start AS VARCHAR))

  SET @start = @start + @step

 END


You can use this via a statement such as:


SELECT 'http://www.mysite.com?page=' + _COUNT from _COUNTER, _MYTABLE WHERE _COUNT <= _TOTAL_PAGES