Learning scrapers, webcrawlers, search engines and CURL

Linux and Unix-Like Linux
1

All,

I'm trying to learn scrapers, webcrawlers, search engines and CURL. I've chosen to interrogate the
following sites:

  • Manta,
  • SuperPages,
  • Yellow Book,
  • Yellow Pages.

These show organizations/businesses by search type/category, so effective in
finding potential clients.

Since I only run Linux, I have the following questions as I consider my approaches:

  • Text only vs regular browser: which is best?
  • wget vs php fileopen vs CURL: Which is best?
  • HTML tag find/parse: Are there libraries that effectively do this?
  • HTML tag find/parse: Is REGEX the best way to parse these? Where are examples?
  • Checking for the new meta-tags of:
<meta name="category" content="">
<meta name="subcategory" content="">

Where the content is the category or subcategory name

EX:

<meta name="category" content="technology">
<meta name="subcategory" content="retail">
<meta name="subcategory" content="electronics">
<meta name="subcategory" content="computers">
<meta name="subcategory" content="laptops">
<meta name="subcategory" content="20+ inch screen">

or

<meta name="category" content="technology">
<meta name="subcategory" content="manufacturer">
<meta name="subcategory" content="electronics">
<meta name="subcategory" content="computers">
<meta name="subcategory" content="laptops">
<meta name="subcategory" content="20+ inch screen">

I'm processing these into a temp DB, but final repository for the data will be
both SugarCRM CE and PHPList.

There is differing information available on each of these at differing sites and
very confusing, as many businesses
will show on all these, but some listing have website and email and others do not
so will be updating existing records with blank fields, when the info is found on
one of the other sites.

Trying to avoid the time consuming process of calling each, however the level of
detail I need for my app I'm writing may still require the calls. I'll see where
I'm at once the intial automation is finished.

I look at all the HOWTOs and they are like GREEk to me, and so not comprehending what I'm
reading. Going to need some help with the comprehension before it all registers and
I come to a basic understanding of both terms and techniques, so sorry to bother you
with a NEWBIE level understanding of this!

Cheers!

OMR/TBNK

2

I think you are better off to get web page content using PHP scripts and parse the files with REGEX.

If you Google around, I am sure you can find many sample PHP scripts that do most of what you want. This is very old technology and there is no need to reinvent the wheel parsing HTML data.

3

Neo, As I stated, still struggling with the terminology and concepts, so patience, I'm total newbie at using this technology, that's why I'm asking Qs as I don't even know where to focus, right now, to accomplish this.
Cheers!
OMR/TBNK

4

Neo,

Think you're right about not re-inventing the wheel! Looking for code now!

Cheers!

PS

Did you get my email?