Web Scraper Plus+ Manual
Here is a PDF version of the manual.
SQL Express Management Studio
Free Microsoft Tool which allows you to run queries on your local PROVIDUSSTD database. Download from Microsoft.
Next Pages Package Template
Copy this package to use as a starting poing for all of your packages! Contains recommended settings like script tag removal and one second bandwidth throttling between requests. Also contains counter datapage template (you should copy this as well and specify database.) Contains steps needed for a typical next pages scrape, with SQL queries included! Please see video tutorials for an in depth explanation of a next pages package.
Chunked package templates
Extracting over 10,000 rows? Copy these packages and refer to this forum post for more info.
Download: Setup Chunked.ppk
Download: Details Chunked.ppk
Web Scraper is heavily-based in SQL. Here is a tutorial to get started with the basics of SQL.
A useful program for sniffing out what forms are acutally submitting. See their website and download the trial version.
A free utility which allows a user to monitor a webpage (the entire page or specific parts of it) at given intervals to see when it is changed. If it is, this can launch a package automatically. (Right-click and choose "Save Target As" to Download)
A command line program that allows you to move all files in a directory (and sub-directories) to a single directory based upon size, filename, and extension. Useful for sorting files once you have crawled them. See a screenshot for its usage. (Right-click and choose "Save Target As" to Download)
Allows you to set your IP address, as well as change a number of other settings. See a screenshot for its usage. (Right-click and choose "Save Target As" to Download)
Zip Code Excel/CSV Sheets
Want to submit information to a form by zip codes? Here is a list of all major US zip codes, along with their respective city and state in Excel and CSV (Comma Separated Values file).
Brute Force 3 Character Dictionary
Allows you to submit all combinations of 3 characters to a form. Excel or CSV
Brute Force 2 Character Dictionary
Same concept, but with 2 characters. Excel or CSV
English Language Dictionary
Huge list of English words, and some combinations with numbers. CSV (too large a file for Excel)
Company Name Permutations
List of thousands of companies. CSV
Here is a web-based counter table that you can scrape any time you need a quick counter table. The table is useful, for example, for cycling through pages. Simply use this URL: http://www.velocityscape.com/Counter.aspx?start=1&end=500&step=1 (feel free to change the numbers) or fill out the following boxes with integers to get the table. And here is the datapage to scrape it.
Alternatively, you can create this table directly in SQL. The following SQL command simply creates an array of numbers between @start and @end increasing by the amount @step. Execute this command in Query Commander or Query Analyzer:
DECLARE @start INTEGER, @end INTEGER, @step INTEGER
SET @start = 1 -- change this to suit your needs
SET @end = 500 -- also change
SET @step = 1 -- also change
CREATE TABLE _COUNTER (_COUNT VARCHAR(20))
WHILE @start <= @end
INSERT INTO _COUNTER (_COUNT)
VALUES (CAST(@start AS VARCHAR))
SET @start = @start + @step
You can use this via a statement such as:
SELECT 'http://www.mysite.com?page=' + _COUNT from _COUNTER, _MYTABLE WHERE _COUNT <= _TOTAL_PAGES