Extract the tables from html

deepti01 · March 14, 2019, 2:27am

Hi I have a script which extracts the table from HTML and convert it into .csv.
But the problem in the script is if we have 2 tables in HTMl . it takes only the first table.

Please help me what changes i need to do in the script to make it read the complete HTML page.

Script is as below:

#!bin/ksh



timestamp=$(date +%d_%m_%y-%T )


export REPORT_PATH=/tefuser5/tef/acw/migwrk1/Informatica/9.5.1/server/infa_shared/Phase4_WLS/bin/reports/COUNT_REPORTS/Work

#mv /tefuser5/tef/acw/migwrk1/Informatica/9.5.1/server/infa_shared/Phase4_WLS/bin/reports/COUNT_REPORTS/Work/ABP_EXP_WRLN.csv /tefuser5/tef/acw/migwrk1/Informatica/9.5.1/server/infa_shared/Phase4_WLS/bin/reports/COUNT_REPORTS/Work/Archive/ABP_EXP_WRLN.csv"_$timestamp"

cd $REPORT_PATH
#rm ABP_EXP_WRLN.csv

#cd $REPORT_PATH
#rm -f *.csv 2>/dev/null

for HTML_F in *.HTML
do
	echo "converting $HTML_F file to csv.."
	dos2unix $HTML_F 1>/dev/null 0>/dev/null 2>/dev/null
 
	l=0
	j=0
	k=0
	rm -f xyz.csv 2>/dev/null
	rm -f abc.csv 2>/dev/null

	while IFS='' read -r line || [[ -n "$line" ]]; 

          # awk '/<TABLE/ {CNT++; if (CNT == 2) P = 1}; P; /<\/TABLE/ {P = 0}'

	do
		

	#    echo "$line"
	#awk '{/<TABLE/}'
    		if [[ "$line" == \<BR\>\<TABLE\ \ width\=* || $j -ge 2 ]]; then
		
			let j=$j+1
			if [ $j -ge 2 ]; then
				#echo "reached in 1st if"
				echo "$line" | grep -i '</TD>' 1>/dev/null 2>/dev/null
				if [ $? -eq 0 ]; then
					#echo "reached in 2nd if"
					tmp=${line#*\"\>}
					#echo "$tmp"
					res=${tmp%%\ \<\/TD\>*}
					echo "$res" >> abc.csv
				else
					:
				fi
			else
				:
			fi
	    	else
				:
    	    	fi

		#echo "$line"
    		if [[ "$line" == *TABLE\>* ]]; then
      			#echo "end of 1st table"
			let k=$k+1
    		fi

    		if [ $k -eq 2 ]; then
			echo "$HTML_F is ending.."
			break
    		fi
	done < "$HTML_F"

		
	
	while read a
	do
		if [ $l -eq 4 ]; then
			l=0
			echo "$a" >> xyz.csv
		else
			let l=$l+1
			echo "$a" | tr -s '\n' ',' >> xyz.csv
		fi
	done < abc.csv

	rm -f abc.csv
	tmpfname=`basename $HTML_F .HTML`
	rm -f $tmpfname.csv 2>/dev/null
	mv xyz.csv $tmpfname.csv

	#printf "\n\n\n\n\n\n\n,,THIS IS END OF FILE,," >> $tmpfname.csv
	#printf "\n\n" >> $tmpfname.csv
	dos2unix $tmpfname.csv 1>/dev/null 0>/dev/null 2>/dev/null
	chr=`echo $tmpfname.csv|cut -d'_' -f1`
	echo "$chr $tmpfname.csv"
	#chkDiff $chr $tmpfname.csv
done


HTML page is as below:


<html>
<body>
<b><br>Running Date: </b>11-JAN-2019 03:07</br>
<h2> Schema mapping and info    </h2>
<BR><TABLE  width="100%" class="x1h" cellpadding="1" cellspacing="0" border="5">
<TR>
<b><td class="x3w" bgcolor="#808080" width="4%"> No </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Exp Schema e </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Export Tables </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Imp Schema </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Import Tables </TD>
<b><td class="x3w" bgcolor="#808080" width="5%"> Diff  </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#E3E4FA">1 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">FVT4 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">54 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">PRDCUSTO </TD>
<b><td class="x3w" bgcolor="#E3E4FA">54 </TD>
<b><td class="x3w" bgcolor="#E3E4FA"> </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#E3E4FA">1 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">FVT4 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">56 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">All Imp Schema</TD>
<b><td class="x3w" bgcolor="#E3E4FA">54 </TD>
<b><td class="x3w" bgcolor="#FF0000">2 </TD></TR>
</TABLE>
<h2> Missing Tables on ImpLogs   </h2>
<h3>       TABLE_NAME :NAME_DATA </h3>
<h3>       TABLE_NAME :WHITE_LIST_MIG </h3>
<h2> Table Rows Comparison   </h2>
<BR><TABLE  width="100%" class="x1h" cellpadding="1" cellspacing="0" border="5">
<TR>
<b><td class="x3w" bgcolor="#808080" width="4%"> No </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> TABLE NAME </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Exported Rows </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Imported Rows </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Diff Rows </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#E3E4FA">1 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">NAME_DATA  </TD>
<b><td class="x3w" bgcolor="#E3E4FA">24760 </TD>
<b><td class="x3w" bgcolor="#E3E4FA"> </TD>
<b><td class="x3w" bgcolor="#FF0000">Not exist on Imp </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#E3E4FA">2 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">WHITE_LIST_MIG  </TD>
<b><td class="x3w" bgcolor="#E3E4FA">12912 </TD>
<b><td class="x3w" bgcolor="#E3E4FA"> </TD>
<b><td class="x3w" bgcolor="#FF0000">Not exist on Imp </TD></TR>
</TABLE>
<h3> Imp and Exp logs Missmatch </h3>
<h2> All Exp , Imp logs Info   </h2>
<BR><TABLE  width="100%" class="x1h" cellpadding="1" cellspacing="0" border="5">
<TR>
<b><td class="x3w" bgcolor="#808080" width="4%"> No  </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> TABLE NAME </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Exported Rows </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Imported Rows </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Diff Rows </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">1 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">ADDRESS_DATA  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">13753 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">13753 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">2 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">ADDRESS_NAME_LINK  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">68715 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">68715 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">3 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AGREEMENT  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">4 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AGREEMENT_RESOURCE  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">29979 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">29979 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">5 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AGR_RES_HISTORY  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">6 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_ACCOUNT  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">12912 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">12912 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">7 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_ADDRESS_NAME  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">25824 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">25824 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">8 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_AGED_TRIAL_BALANCE  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">18780 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">18780 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">9 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_BILLING_ARRANGEMENT  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">12912 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">12912 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">10 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_CHARGES  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">18069 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">18069 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">11 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_CHARGE_GROUP  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">18069 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">18069 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">12 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_CREDIT_DEBIT_LINK  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">11032 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">11032 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">13 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_CUSTOMER_CREDIT  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">359 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">359 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">14 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_INVOICE  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">18428 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">18428 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">15 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_JGL_CONTROL  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">16 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_PAYMENT  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">8439 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">8439 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">17 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_PAYMENT_DETAILS  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">8439 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">8439 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">18 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_PAY_CHANNEL  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">12912 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">12912 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">19 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_PROOF_AND_BALANCE  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">4 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">4 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">20 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_TAX_ITEM  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">21 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_TRANSACTION_LOG  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">26867 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">26867 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">22 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_UNAPPLIED_CREDIT  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">711 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">711 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">23 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_ACTIVITY_HISTORY  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">30928 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">30928 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">24 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_BILL_STATEMENT  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">17269 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">17269 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">25 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_BLNG_ARRANGEMENT  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">12912 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">12912 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">26 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_CHARGE  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">17269 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">17269 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">27 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_CHARGE_REQUEST  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">1966 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">1966 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">28 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_CUSTOMER  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">12912 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">12912 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">29 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_CUSTOMER_INFO  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">55803 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">55803 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">30 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_CYCLE_CUSTOMERS  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">17269 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">17269 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">31 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_CYC_PAYER_POP  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">17269 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">17269 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">32 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_DOCUMENT  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">17269 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">17269 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">33 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_INVOICE  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">17269 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">17269 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">34 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_INV_CHARGE_REL  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">17269 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">17269 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">35 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_PAY_CHANNEL  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">12912 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">12912 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">36 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_RC_RATES  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">30928 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">30928 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">37 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_TAX  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">38 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL1_TAX_ITEM  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">39 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">BL9_PROVINCIAL_PCP  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">18259 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">18259 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>

RavinderSingh13 · March 14, 2019, 2:47am

Hello deepti01,

Good that you have shown us your code what you have tried but for guidance IMHO you should keep your question to the point and crispy.
Kindly do let us know complete requirement along with that show us small samples of input and expected output in CODE TAGS(see how I edited your code into tags and it is looking different now) and let us know then.

Thanks,
R. Singh

deepti01 · March 14, 2019, 2:54am

Hi ,

I have reduced the html code :
Below you can see two tables ,but my code is picking just one table.
I need to know what modification i need to do in the code so that it may take the complete html tables (both tables)

<html>
<body>
<b><br>Running Date: </b>11-JAN-2019 03:07</br>
<h2> Schema mapping and info    </h2>
<BR><TABLE  width="100%" class="x1h" cellpadding="1" cellspacing="0" border="5">
<TR>
<b><td class="x3w" bgcolor="#808080" width="4%"> No </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Exp Schema e </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Export Tables </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Imp Schema </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Import Tables </TD>
<b><td class="x3w" bgcolor="#808080" width="5%"> Diff  </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#E3E4FA">1 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">FVT4 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">54 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">PRDCUSTO </TD>
<b><td class="x3w" bgcolor="#E3E4FA">54 </TD>
<b><td class="x3w" bgcolor="#E3E4FA"> </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#E3E4FA">1 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">FVT4 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">56 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">All Imp Schema</TD>
<b><td class="x3w" bgcolor="#E3E4FA">54 </TD>
<b><td class="x3w" bgcolor="#FF0000">2 </TD></TR>
</TABLE>
<h2> Missing Tables on ImpLogs   </h2>
<h3>       TABLE_NAME :NAME_DATA </h3>
<h3>       TABLE_NAME :WHITE_LIST_MIG </h3>
<h2> Table Rows Comparison   </h2>
<BR><TABLE  width="100%" class="x1h" cellpadding="1" cellspacing="0" border="5">
<TR>
<b><td class="x3w" bgcolor="#808080" width="4%"> No </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> TABLE NAME </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Exported Rows </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Imported Rows </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Diff Rows </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#E3E4FA">1 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">NAME_DATA  </TD>
<b><td class="x3w" bgcolor="#E3E4FA">24760 </TD>
<b><td class="x3w" bgcolor="#E3E4FA"> </TD>
<b><td class="x3w" bgcolor="#FF0000">Not exist on Imp </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#E3E4FA">2 </TD>
<b><td class="x3w" bgcolor="#E3E4FA">WHITE_LIST_MIG  </TD>
<b><td class="x3w" bgcolor="#E3E4FA">12912 </TD>
<b><td class="x3w" bgcolor="#E3E4FA"> </TD>
<b><td class="x3w" bgcolor="#FF0000">Not exist on Imp </TD></TR>
</TABLE>
<h3> Imp and Exp logs Missmatch </h3>
<h2> All Exp , Imp logs Info   </h2>
<BR><TABLE  width="100%" class="x1h" cellpadding="1" cellspacing="0" border="5">
<TR>
<b><td class="x3w" bgcolor="#808080" width="4%"> No  </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> TABLE NAME </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Exported Rows </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Imported Rows </TD>
<b><td class="x3w" bgcolor="#808080" width="20%"> Diff Rows </TD></TR>
<tr>
<b><td class="x3w" bgcolor="#BDD7EE">1 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">ADDRESS_DATA  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">13753 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">13753 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>

Neo · March 14, 2019, 3:40am

Thanks for adding your code.

Is this homework?

If not homework, what are you actually trying to accomplish?

Thanks.

deepti01 · March 14, 2019, 3:51am

I need to get the complete HTML tables into the .csv file which is getting created through the script.

But the issue in the script is , it is just reading the first table that has <TABLE> </TABLE> tags . I need the script to also read and extract the data from the second table which also has <TABLE></TABLE> tag.

Script is already attached.

Thanks,
Deepti

Neo · March 14, 2019, 3:59am

Is this homework?

It sure looks like homework from school to me for many reasons.

deepti01 · March 14, 2019, 4:01am

i am just learning to do it . can you help please.

RudiC · March 14, 2019, 4:41am

Doesn't really answer Neo's question. Please explicitly tell us HOW and WHERE you learn to do it. What is your motivation to do so?

I see you incorporated but commented out my recent proposal to extract the second table from the HTML file. What keeps you from adapting it to your needs?

And, why don't you go for doing all of your task in one single script, e.g. awk ?

RavinderSingh13 asked you to post "samples of input and expected output". If done, you could get hints on how to do it optimally.

deepti01 · March 14, 2019, 4:55am

Hi,

The end use if this task is to dump the data into the database through Informatica.
So for that I am converting the HTML files to .csv file.

Sample input : i have already placed the html code . that is my sample file (.html)

Sample output should be like:

No    TABLE NAME    Exported Rows    Imported Rows    Diff Rows
1    NAME_DATA    24760        Not exist on Imp
2    WHITE_LIST_MIG    12912        Not exist on Imp
1    ADDRESS_DATA    13753    13753    0
2    ADDRESS_NAME_LINK    68715    68715    0
3    AGREEMENT    0    0    0
4    AGREEMENT_RESOURCE    29979    29979    0
5    AGR_RES_HISTORY    0    0    0
 6    AR1_ACCOUNT    12912    12912    0

RudiC · March 14, 2019, 5:33am

Please note that awk is NOT an adequate tool (nor is shell) to analyse / extract html data; there are way more apt tools to do this.
Please note as well that none of your files is valid html.
And, "csv" means "comma separated values", so I used commas for the extracted datas' separator.

Howsoever, for this - and exactly this - simple case, try

awk '
/<TABLE/        {P = 1
                }
/<\/TABLE/      {P = 0
                }
!P              {next
                }
/<[Tt][Rr]>/    {printf ORS
                 DL = ""
                }
/<td/           {gsub (/<[^>]*>/, "")
                 gsub (/^ *| *$/, "")
                 printf "%s%s", DL, $0
                 DL = ","
                }
' file

No,ExpSchemae,ExportTables,ImpSchema,ImportTables,Diff
1,FVT4,54,PRDCUSTO,54,
1,FVT4,56,AllImpSchema,54,2
No,TABLENAME,ExportedRows,ImportedRows,DiffRows
1,NAME_DATA,24760,,NotexistonImp
2,WHITE_LIST_MIG,12912,,NotexistonImp
No,TABLENAME,ExportedRows,ImportedRows,DiffRows
1,ADDRESS_DATA,13753,13753,0
2,ADDRESS_NAME_LINK,68715,68715,0
3,AGREEMENT,0,0,0
4,AGREEMENT_RESOURCE,29979,29979,0
5,AGR_RES_HISTORY,0,0,0
6,AR1_ACCOUNT,12912,12912,0
.
.
.

drl · March 14, 2019, 1:00pm

Hi.

There are some codes that understand HTML, such as lynx , elinks . For example, here is elinks extracting all the stuff in the sample HTML, and then, step-by-step, tables are refined, with sed, grep, etc., into a CSV file:

#!/usr/bin/env bash

# @(#) s1       Demonstrate table extraction from HTML.

# Utility functions: print-as-echo, print-line-with-visual-space, debug.
# export PATH="/usr/local/bin:/usr/bin:/bin"
LC_ALL=C ; LANG=C ; export LC_ALL LANG
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
em() { pe "$*" >&2 ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [ -f $C ] && $C specimen elinks grep dixf

FILE=${1-data1}

# Utility functions: print-as-echo, print-line-with-visual-space.
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }

pl " Input data file $FILE:"
specimen 5:5:5 $FILE

pl " Results:"
elinks -dump $FILE |
tee t1 |
grep -v '^  *[|+-]-' |
tee t2 |
grep '^  *[|]' |
tee t3 |
sed -e 's/^  *[|]//' -e 's/[|]$//' |
tee t4 |
sed 's/  *[|]/,/g' |
tee t5 |
specimen 5:5:5

pl " Samples of intermediate output:"
head -5 t[1-5]

pl " Some details for codes:"
dixf elinks specimen grep

exit 0

producing:

$ ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 3.16.0-7-amd64, x86_64
Distribution        : Debian 8.11 (jessie) 
bash GNU bash 4.3.30
specimen (local) 1.17
ELinks 0.12pre6
grep (GNU grep) 2.20
dixf (local) 1.57

-----
 Input data file data1:
Edges: 5:5:5 of 295 lines in file "data1"
<html>
<body>
<b><br>Running Date: </b>11-JAN-2019 03:07</br>
<h2> Schema mapping and info    </h2>
<BR><TABLE  width="100%" class="x1h" cellpadding="1" cellspacing="0" border="5">
   ---
<b><td class="x3w" bgcolor="#BDD7EE">15 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">AR1_JGL_CONTROL  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
   ---
<b><td class="x3w" bgcolor="#BDD7EE">BL9_PROVINCIAL_PCP  </TD>
<b><td class="x3w" bgcolor="#BDD7EE">18259 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">18259 </TD>
<b><td class="x3w" bgcolor="#BDD7EE">0 </TD></TR>
<tr>

-----
 Results:
Edges: 5:5:5 of 46 lines in file "-"
No,Exp Schema e,Export Tables,Imp Schema,Import Tables,Diff
1,FVT4,54,PRDCUSTO,54,    
1,FVT4,56,All Imp Schema,54,2   
No,TABLE NAME,Exported Rows,Imported Rows,Diff Rows        
1,NAME_DATA,24760,,Not exist on Imp 
   ---
15,AR1_JGL_CONTROL,0,0,0             
16,AR1_PAYMENT,8439,8439,0             
17,AR1_PAYMENT_DETAILS,8439,8439,0             
18,AR1_PAY_CHANNEL,12912,12912,0             
19,AR1_PROOF_AND_BALANCE,4,4,0             
   ---
35,BL1_PAY_CHANNEL,12912,12912,0             
36,BL1_RC_RATES,30928,30928,0             
37,BL1_TAX,0,0,0             
38,BL1_TAX_ITEM,0,0,0             
39,BL9_PROVINCIAL_PCP,18259,18259,0             

-----
 Samples of intermediate output:
==> t1 <==
   Running Date: 11-JAN-2019 03:07

Schema mapping and info

   +------------------------------------------------------------------------+

==> t2 <==
   Running Date: 11-JAN-2019 03:07

Schema mapping and info

   |No |Exp Schema e   |Export Tables  |Imp Schema     |Import Tables  |Diff|

==> t3 <==
   |No |Exp Schema e   |Export Tables  |Imp Schema     |Import Tables  |Diff|
   |1  |FVT4           |54             |PRDCUSTO       |54             |    |
   |1  |FVT4           |56             |All Imp Schema |54             |2   |
   |No  |TABLE NAME      |Exported Rows   |Imported Rows  |Diff Rows        |
   |1   |NAME_DATA       |24760           |               |Not exist on Imp |

==> t4 <==
No |Exp Schema e   |Export Tables  |Imp Schema     |Import Tables  |Diff
1  |FVT4           |54             |PRDCUSTO       |54             |    
1  |FVT4           |56             |All Imp Schema |54             |2   
No  |TABLE NAME      |Exported Rows   |Imported Rows  |Diff Rows        
1   |NAME_DATA       |24760           |               |Not exist on Imp 

==> t5 <==
No,Exp Schema e,Export Tables,Imp Schema,Import Tables,Diff
1,FVT4,54,PRDCUSTO,54,    
1,FVT4,56,All Imp Schema,54,2   
No,TABLE NAME,Exported Rows,Imported Rows,Diff Rows        
1,NAME_DATA,24760,,Not exist on Imp 

-----
 Some details for codes:

elinks  lynx-like alternative character mode WWW browser (man)
Path    : /usr/bin/elinks
Version : 0.12pre6
Type    : ELF 64-bit LSB shared object, x86-64, version 1 ( ...)
Repo    : Debian 8.11 (jessie) 
Home    : http://elinks.cz/ (pm)

specimen        Extract, display sections, sample, provide taste of file. (what)
Path    : ~/bin/specimen
Version : 1.17
Length  : 334 lines
Type    : Perl script, ASCII text executable
Shebang : #!/usr/bin/env perl
Modules : (for perl codes)
 warnings       1.23
 strict 1.08
 English        1.09
 Carp   1.3301
 Data::Dumper   2.151_01
 Getopt::Long   2.42
 feature        1.36_01
 Regexp::Common 2013031301
 Algorithms::Numerical::Sample  1.1

grep    print lines matching a pattern (man)
Path    : /bin/grep
Version : 2.20
Type    : ELF 64-bit LSB shared object, x86-64, version 1 ( ...)
Repo    : Debian 8.11 (jessie) 
Home    : http://www.gnu.org/software/grep/ (pm)

The intermediate sample show the transformations that are done, and could, of course, probably be collected into an awk or perl script.

Best wishes ... cheers, drl