What is Best Way to Automate Download and Processing of Large Amounts of Data from a Web Site?

Questions & Suggestions
5

Here's a small update:
From this recommendation I installed SQL Pro Studio and created an SQlite database with two tables "QSDBold" and "QSDBnew". With a query which I found on stackoverflow, it is possible to filter the unique identifier (column 1) like this:

Listing the lines which have been removed:

SELECT DISTINCT "Proben-ID"
FROM QSDBold
WHERE "Proben-ID" Not IN 
(SELECT DISTINCT "Proben-ID" FROM QSDBnew)

And by switching new and old in the query, it will list the lines that are new:

SELECT DISTINCT "Proben-ID"
FROM QSDBnew
WHERE "Proben-ID" Not IN 
(SELECT DISTINCT "Proben-ID" FROM QSDBold)

Now, I am just confused how to get KM and SQL Studio to work together. Can SQL Studio be scripted? Do I need to use interface scripting with KM? Is there a better solution?