Noob question about parsing a website

UNIX for Beginners Q & A UNIX for Dummies Questions & Answers
1

I'm trying to parse the website, General Electric Company (GE) Stock Price, News, Quote & History - Yahoo Finance, and retrieve the info between <span id="yfs_l84_ge">18.98</span>, so 18.98.

What would be the best way to go about this in a bash script?

Any help or suggestions will be much appreciated.
Thanks!

2

If bash script weren't a stringent requirement, I would suggest python, perl or php. Python has a package called BeautifulSoup to do exactly this... (you will need to also import urllib for fetching the page). Google-ing for these could give you some ready made scripts as well.

For bash script, you will need to use curl and awk (alternatively use wget and grep) but will be time consuming to get it done right (while soup.find("span", {'id' : "yfs_l84_ge"}) within your python parser would get you your required element).

Here's a quick non-debugged python code:

from BeautifulSoup import BeautifulSoup
import urllib
import urllib2

url = "<your website to crawl>"
user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)'

values = {'name' : 'Friendly Spider',
          'location' : 'New York, USA',
          'language' : 'Python'}

headers = {'User-Agent' : user_agent }
urlData = urllib.urlencode(values)
req = urllib2.Request(url, urlData, headers)
response = urllib2.urlopen(url)
the_page = response.read()
soup = BeautifulSoup(the_page)

span = soup.find("span", {'id' : "yfs_l84_ge"})
if span:
   print span.contents[0].strip()
3

Try awk:

awk -F\> '/^span id="yfs_l84_ge"/{print $2}' RS=\<

awk '$1==s{print $2}' RS=\< FS=\> s='span id="yfs_l84_ge"'

awk '$1=="span id=\"" i "\""{print $2}' RS=\< FS=\> i="yfs_l84_ge"