Adding incomplete HTML code to a file

Shell Programming and Scripting
1

Hi folks,

I am scraping data from the Internet that has the format similar to what's on this page -- Trigger Notice Report

The code I've written for scraping and storing results works fine when the HTML code is well written, but not when there are mistakes. In particular, the code breaks when there are no tags for closing a table rows.

If you view the source for the link above, you will see that there are '</TR>'s which indicate the end of an HTML row, but sometimes not a new '<TR>' to indicate the beginning of the next row.

I need to use somethink like awk or sed to basically do the following -- insert a line with ''<TR>'' whenever the previous line was "</TR>" followed by a ''<TD [some text]''. For example, in the code below, I need a line with ''<TR>'' just before the highlighted line. The rest of the HTML file follows pretty much the same pattern. Any suggestions?

<TR bgcolor="#C0C0C0">
<TD headers='Arizona noinfo' align=center width=25 ><FONT size='-2'>@</FONT></TD>
<TD headers='Arizona noinfo' align=center  width=25><FONT size='-2'>�</FONT></TD>
<TD headers='Arizona noinfo' align=center  width=25><FONT size='-2'>�</FONT></TD>
<TH id='Arizona' align=left  width=100><FONT size='-2'>Arizona</FONT></TH>
<TD headers='Arizona noinfo' align=center  width=50><FONT size='-2'>2</FONT></TD>
<TD headers='Arizona noinfo' align=center  width=50><FONT size='-2'>2</FONT></TD>
<TD headers='Arizona 13_week_IUR indicators' align=center  width=50><FONT size='-2'>2.74</FONT></TD>
<TD headers='Arizona pct_of_prior_2years indicators' align=center  width=50><FONT size='-2'>86</FONT></TD>
<TD headers='Arizona 3_mo_satur indicators' align=center  width=50><FONT size='-2'>9.4</FONT></TD>
<TD headers='Arizona year pct_of_prior indicators' align=center  width=50><FONT size='-2'>102</FONT></TD>
<TD headers='Arizona 2nd_year pct_of_prior indicators' align=center  width=50><FONT size='-2'>128</FONT></TD>
<TD headers='Arizona 2nd_year pct_of_prior indicators' align=center  width=50><FONT size='-2'>223</FONT></TD>
<TD headers='Arizona avail_wks pct_of_prior indicators noinfo' align=center width=50><FONT size='-2'>20</FONT></TD>
<TD headers='Arizona dates periods status' align=center width=100><FONT size='-2'>B 02-22-2009</FONT></TD>
</TR>


<TD headers='Arkansas noinfo' align=center width=25 ><FONT size='-2'>�</FONT></TD>
<TD headers='Arkansas noinfo' align=center  width=25><FONT size='-2'>�</FONT></TD>
<TD headers='Arkansas noinfo' align=center  width=25><FONT size='-2'>&</FONT></TD>
<TH id='Arkansas' align=left  width=100><FONT size='-2'>Arkansas</FONT></TH>
<TD headers='Arkansas noinfo' align=center  width=50><FONT size='-2'>2</FONT></TD>
<TD headers='Arkansas noinfo' align=center  width=50><FONT size='-2'>2</FONT></TD>
<TD headers='Arkansas 13_week_IUR indicators' align=center  width=50><FONT size='-2'>4.02</FONT></TD>
<TD headers='Arkansas pct_of_prior_2years indicators' align=center  width=50><FONT size='-2'>87</FONT></TD>
<TD headers='Arkansas 3_mo_satur indicators' align=center  width=50><FONT size='-2'>7.9</FONT></TD>
<TD headers='Arkansas year pct_of_prior indicators' align=center  width=50><FONT size='-2'>103</FONT></TD>
<TD headers='Arkansas 2nd_year pct_of_prior indicators' align=center  width=50><FONT size='-2'>131</FONT></TD>
<TD headers='Arkansas 2nd_year pct_of_prior indicators' align=center  width=50><FONT size='-2'>154</FONT></TD>
<TD headers='Arkansas avail_wks pct_of_prior indicators noinfo' align=center width=50><FONT size='-2'>�</FONT></TD>
<TD headers='Arkansas dates periods status' align=center width=100><FONT size='-2'>E 09-26-2009</FONT></TD>
</TR>
2

Try this:

awk 'BEGIN{FS=">";OFS=FS}
   /<TR/ {r++}
   /<\/TR/ {r--}
   /<TD/&&!r{print "<TR>";r--} 1' infile
3

I think this should take care of neatly arranged HTML files as you presented, and more
realistic ones where </TR> might not be on a record by itself.

awk '
    /<\/tr>$/ {
        print;
        getline;
        if( substr( $1, 1, 3 ) == "<td" )
            printf( "<tr>%s\n", $0 );
        next;
    }
    {
        gsub( "<\/tr>[[:space:]]*<td", "</tr><tr><td " );
        print;
    }
' IGNORECASE=1 input-file >output-file

EDIT: @Chubler_XL -- nice.

4

Hey Chubler_XL and agama,

Thanks for your help! Maybe I'm doing something wrong, but I tried running agama's code but it doesn't seem to be doing anything to the file (weird). I tried Chubler's and it fixed only the first instance, but not all instances of the issue.

Any suggestions?

5

try this

awk '/TR/{if($1 ~/^<TR/){f=1};if($1 ~ /^<\/TR/){f=0};print;next}(f==0&&NF>0){print "<TR sometext>";f=1}{print}' filename