Scrape 10 million pages and save the raw html data in mysql database

Shell Programming and Scripting
1

I have a list of 10 million page urls. I want those pages scraped and saved in the mysql database as raw html.

I own a Linux VPS server with 1GB RAM and WHM/cPanel.

I would like to scrape at least 100,000 urls in 24 hours.

So can anyone give me some sample shell scripting code?

2

Why do you need to scrape 10 million urls?

Regards,
Alister

3

To collect information.

4

Well, that explains everything. I now feel confident that there is nothing shady afoot.

Regards,
Alister

5

If you are not willing to help, whats the point of leaving reply to my thread.?