How to extract url from html page?

Shell Programming and Scripting
1

for example, I have an html file, contain

<a href="http://awebsite"  id="awebsite" class="first">website</a>

and sometime a line contains more then one link, for example

<a href="http://awebsite"  id="awebsite" class="first">website</a><a href="http://bwebsite"  id="bwebsite" class="first">websiteb</a>

how can I extract that become something like this.

http://awebsite website
http://bwebsite websiteb

I only know how to get word between <a>and</a> using

sed -e 's/<[^<]*>//g'

but I don't know how to get the link..
thanks

2 
#!/usr/bin/env ruby -Ku

require 'hpricot'
doc = open("file"){|f|Hpricot(f)}
(doc/"a").each do |x|
 print "-->#{x.get_attribute("href)}, #{x.inner_text}\n"
end



$ ruby geturl.rb
-->http://awebsite, website
-->http://bwebsite, websiteb
3

thanks for the answering, but I don't really know what is hpricot and how to use it.
I'm new to shell programming and I don't know ruby at all.:frowning:

do you know any solution using sed, awk, or grep maybe?
thanks.

4

Try...

 
awk -F'href="|"  |">|</' '{for(i=2;i<=NF;i=i+4) print $i,$(i+2)}' infile
5 
$ cat file
<a href="http://awebsite"  id="awebsite" class="first" someattribute="last" > website</a>
<a href="http://bwebsite"  id="bwebsite" class="first">websiteb</a>

$ awk -F'href="|"  |">|</' '{for(i=2;i<=NF;i=i+4) print $i,$(i+2)}' file
http://awebsite a>
http://bwebsite websiteb

$ ruby test.rb
-->http://awebsite,  website
-->http://bwebsite, websiteb
6 
grep -o 'http://[^"]*'
7

OP also needs the inner text, not just urls.

8

Oops...

awk '/^id/{print $2}/^href/{printf $2 RS}' FS='"' RS=" " infile
9 
<a href="http://awebsite"  id="awebsite" class="first" someattribute="last" > website</a>
<a href="http://bwebsite"  id="bwebsite" class="first">websiteb</a>

Nice one but I guess the OP needs the one which is in red only, not $2 after id.

---------- Post updated at 12:53 PM ---------- Previous update was at 12:45 PM ----------

I guess no space before and after >(on red below) in the original requirement from the OP so it will work fine as below if you remove the space

 
$ cat file
<a href="http://awebsite"  id="awebsite" class="first" someattribute="last" > website</a>
<a href="http://bwebsite"  id="bwebsite" class="first">websiteb</a>
 
$ awk -F'href="|"  |">|</' '{for(i=2;i<=NF;i=i+4) print $i,$(i+2)}' file
http://awebsite  website
http://bwebsite websiteb
10

HTML is not regular. you won't know in future there will be more or less spaces as web page changes. Ideally, we should use a parser.

11

Wow I am really having trouble reading today... This then maybe?

awk '$2=="a href="{printf $3; getline; print OFS $1}' RS='>' FS='["<]'
12

Nice. works with OP's mini sample. But I used it to parse a page like that of Google, for exampl, i hit error.

awk: (FILENAME=file FNR=239) fatal: not enough arguments to satisfy format string
13

OK :slight_smile: , the next attempt then:

awk -F'>' '/^a href/{split($1,F,"\"");print F[2],$NF}' RS='<'
14

Nice. It worked. Now, how about taking care of HTML entities like ?
This is a symbol (down arrow). Gawk doesn't return it. But my parser does. Any ways to set it with gawk?

15

Do you have a sample url?

16

Its just Google. Google is just an example.

17

When I do wget www.google.com , I see no -character.

18

ok, the first time i do my test, i copied the html using the browser's view source. But since you mentioned wget, so here's how i do my next test. Using wget to download Google

$ wget 209.85.132.104

$ awk -F'>' '/^a href/{split($1,F,"\"");print F[2],$NF}' RS='<' index.html
http://mail.google.com/mail/?hl=en&tab=wm Gmail
http://www.google.com/intl/en/options/
/url?sa=p&pref=ig&pval=3&q=http://www.google.com/ig%3Fhl%3Den%26source%3Diglk&usg=AFQjCNFA18XPfgb7dKnXfKz7x7g1GDH1tg iGoogle
/preferences?hl=en Settings
https://www.google.com/accounts/Login?hl=en&continue=http://209.85.132.104/ Sign in
/advanced_search?hl=en Advanced Search
/language_tools?hl=en Language Tools
/intl/en/ads/ Advertising�Programs
/services/ Business Solutions
/intl/en/about.html About Google
http://www.google.com/ncr Go to Google.com
/intl/en/privacy.html Privacy


$ ruby test.rb
-->http://www.google.com/imghp?hl=en&tab=wi, Images
-->http://video.google.com/?hl=en&tab=wv, Videos
-->http://maps.google.com/maps?hl=en&tab=wl, Maps
-->http://news.google.com/nwshp?hl=en&tab=wn, News
-->http://www.google.com/prdhp?hl=en&tab=wf, Shopping
-->http://mail.google.com/mail/?hl=en&tab=wm, Gmail
-->http://www.google.com/intl/en/options/, more �
-->/url?sa=p&pref=ig&pval=3&q=http://www.google.com/ig%3Fhl%3Den%26source%3Diglk&usg=AFQjCNFA18XPfgb7dKnXfKz7x7g1GDH1tg, iGoogle
-->/preferences?hl=en, Settings
-->https://www.google.com/accounts/Login?hl=en&continue=http://209.85.132.104/, Sign in
-->/advanced_search?hl=en, Advanced Search
-->/language_tools?hl=en, Language Tools
-->/intl/en/ads/, Advertising*Programs
-->/services/, Business Solutions
-->/intl/en/about.html, About Google
-->http://www.google.com/ncr, Go to Google.com
-->/intl/en/privacy.html, Privacy

If you notice at Google main page, there is a link called "more" right at the top, and the down arrow key is next to it, which is reflected in the ruby output as

-->http://www.google.com/intl/en/options/, more �

The down arrow key is itself a url link.

1 Like
19

Thanks Kurumi, it was because of event definitions in the tag.

awk -F'>' '/[ \t]href=/{N=split($1,F,"\""); i=1; while(F[i++]!~/[ \t]href=/); print F,$NF}' RS='<'
1 Like
20

thanks scrutinizer for following up on this. you are almost there. :slight_smile:

$ awk -F'>' '/[ \t]href=/{N=split($1,F,"\""); i=1; while(F[i++]!~/[ \t]href=/); print F,$NF}' RS='<' index.html |sort
/advanced_search?hl=en Advanced Search
http://mail.google.com/mail/?hl=en&tab=wm Gmail
http://maps.google.com/maps?hl=en&tab=wl Maps
http://news.google.com/nwshp?hl=en&tab=wn News
https://www.google.com/accounts/Login?hl=en&continue=http://209.85.132.104/ Sign in
http://video.google.com/?hl=en&tab=wv Videos
http://www.google.com/imghp?hl=en&tab=wi Images
http://www.google.com/intl/en/options/
http://www.google.com/ncr Go to Google.com
http://www.google.com/prdhp?hl=en&tab=wf Shopping
/intl/en/about.html About Google
/intl/en/ads/ Advertising�Programs
/intl/en/privacy.html Privacy
/language_tools?hl=en Language Tools
/preferences?hl=en Settings
/services/ Business Solutions
/url?sa=p&pref=ig&pval=3&q=http://www.google.com/ig%3Fhl%3Den%26source%3Diglk&usg=AFQjCNFA18XPfgb7dKnXfKz7x7g1GDH1tg iGoogle

$ ruby test.rb index.html |sort
-->/advanced_search?hl=en, Advanced Search
-->http://mail.google.com/mail/?hl=en&tab=wm, Gmail
-->http://maps.google.com/maps?hl=en&tab=wl, Maps
-->http://news.google.com/nwshp?hl=en&tab=wn, News
-->https://www.google.com/accounts/Login?hl=en&continue=http://209.85.132.104/, Sign in
-->http://video.google.com/?hl=en&tab=wv, Videos
-->http://www.google.com/imghp?hl=en&tab=wi, Images
-->http://www.google.com/intl/en/options/, more �
-->http://www.google.com/ncr, Go to Google.com
-->http://www.google.com/prdhp?hl=en&tab=wf, Shopping
-->/intl/en/about.html, About Google
-->/intl/en/ads/, Advertising*Programs
-->/intl/en/privacy.html, Privacy
-->/language_tools?hl=en, Language Tools
-->/preferences?hl=en, Settings
-->/services/, Business Solutions
-->/url?sa=p&pref=ig&pval=3&q=http://www.google.com/ig%3Fhl%3Den%26source%3Diglk&usg=AFQjCNFA18XPfgb7dKnXfKz7x7g1GDH1tg, iGoogle