COUNTER SUSHI harvester, pt 4: Using Sushi Py

My SUSHI client for harvesting COUNTER reports is in a relatively stable condition at the moment, so now it's time to share.

How to Use sushi_py

1. Make sure you've read my previous post on setting up your environment. 
2. Download sushi_py.zip or sushi_py.tar.gz and unzip to a location of your choice.
3. Set up your CSV file and/or MySQL tables. Fill them with parameters for all the SUSHI reports you want to regularly schedule. You can find more about how each vendor handles the parameters at the SUSHI server registry.
   
   • For CSV: Open sushi.csv and fill the rows with parameters for your SUSHI requests. Note that sushi_py assumes you have column headers in the first row and will ignore any data you enter in it.
   • For MySQL: Download and run this SQL script that will automatically generate the tables used by sushi_py. If you are using MySQL Workbench, open your database in query mode and go to File > Open SQL Script. Fill the table counter_sushi with the parameters for your SUSHI requests.
4. The file counter_harvest.py is an example of how to use sushi_py. The easiest way to use it is to put it in the same directory as sushi_py.py (unless you want to fiddle with paths and modules). Open counter_harvest.py in a text editor. Edit the MySQL database variables and the report locations. By default, the script is set up to use MySQL. To read SUSHI parameters from a CSV file, uncomment the line with "sushi.GetParamsCSV" and make sure it's pointing to the file you use for your SUSHI parameters. Then, comment out the "sushi.GetParamsSQL" line. To write to CSV, uncomment the line with "sushi.WriteResponseCSV", and comment out the line with "sushi.WriteResponseSQL".
5. Run counter_harvest.py. I recommend you run it in a command line so that you can view the printout of which reports worked and which failed. To enable debug mode, uncomment or add the line "sushi.DebugMode = True". Warning: debug mode will crank out _a lot_ of text. Consider running just one report at a time for debug mode. You can run one report at a time easily by modifying the for loop in counter_harvest.py so that it only runs CallServer and WriteResponse if i == x, where x is the row in your parameter list containing the report you want to test.
6. Once you are comfortable that everything is working, set up counter_harvest.py as a Linux cron job or a Windows scheduled task.

For now, sushi_py works for JR1, DB1, DB2, and DB3 COUNTER reports. It can be easily expanded to work with other reports, but I don't have access to any SUSHI services that allow other reports to test them.

In developing Sushi Py, I've ran into a handful of problems with individual vendor implementations. At the time of writing, Sushi Py has been confirmed to work with the following vendors:

• American Chemical Society
• EBSCO
• Ovid

It has been confirmed _not_ to work with the following:

• Cambridge
• OCLC
• ProQuest

For the three that are not working, I am in contact with technical support to determine the cause. I am fairly sure that it is a problem with their SUSHI implementation because I am not even able to make a successful request using SoapUI. Hopefully, they will be working shortly.

For the most part, Sushi Py does what I need it to do for now, and I will mostly be updating the code on an as-needed basis. Feel free to contact me with questions.