Extract strings from the file using awk s

UNIX for Beginners Q & A
1

Hi,

I'm trying to make a file that gives me useful information. Format of file is below:-

--- (Tue Apr 14 09:46:43 EDT 2020): JOIN_Constraints_Schema:test_simple_joins -------------
internal optimizer errors: 0
--- (Tue Apr 14 09:48:10 EDT 2020): JOIN_Constraints_Schema:test_constraint_joins_setop_dis_oby_lmt -------------
Number of queries that causes internal optimizer errors: 0
--- (Tue Apr 14 09:49:02 EDT 2020): in External_Table_Schema:test_subquery_in_from ---------------
--- (Tue Apr 14 09:49:10 EDT 2020): EventSeries_Schema:test_Event_Series1 -------------
--- (Tue Apr 14 09:49:17 EDT 2020):  Gosalesdw_Schema:test_complex_analytics -------------
--- (Tue Apr 14 09:49:25 EDT 2020):  GBY_Schema1:test_Groupby_Rollup -------------
internal optimizer errors: 0
--- (Tue Apr 14 09:49:40 EDT 2020):  GBY_Schema1:test_Groupby_GroupingSets -------------
internal optimizer errors: 0
--- (Tue Apr 14 09:49:52 EDT 2020):  GBY_Schema1:test_Groupby_Cube -------------
internal optimizer errors: 0
--- (Tue Apr 14 09:50:05 EDT 2020):  GBY_Schema1:test_gby -------------
internal optimizer errors: 0

I need to extract the text after the timestamp and immediately next line error number having internal optimizer error.

It should be like - newout.txt

JOIN_Constraints_Schema,test_simple_joins,0
JOIN_Constraints_Schema,test_constraint_joins_setop_dis_oby_lmt,0
GBY_Schema1,test_Groupby_Rollup,0
GBY_Schema1,test_Groupby_GroupingSets,0
GBY_Schema1,test_Groupby_Cube,0
GBY_Schema1,test_gby,0

Exception - If there is no internal optimizer error attached to any field given in timestamp then no need to print

I tried using a simple script but not works for me

cat newout.txt | while read x
do
        if [ $(echo $x | grep -E '20[0-9][0-9]\):' | wc -l) == 1 ]
        then
                 schema=$(echo $x | awk '{print $(NF-1)}' | awk -F ':' '{print $1}')
                 config=$(echo $x | awk '{print $(NF-1)}' | awk -F ':' '{print $2}')
         else
                 internal=$(echo $x | awk -F ':' '{print $2}')
         fi
         echo $schema,$config,$internal
done

Please help me achieve this.

2

something along these lines - a bit verbose: awk -f mannu.awk newout.txt where mannu.awk is:

BEGIN {
  FS=":"
  OFS=","
  tab=sprintf("\t")
}
function trim(str)
{
    sub("^([ ]*|" tab "*)", "", str);
    sub("([ ]*|" tab "*)" "$", "", str);
    return str;
}

$1~/^---/ {m=trim($4) OFS $5;m=substr(m,1,index(m," ")-1)}
/internal optimizer errors:/ {print m,$NF+0}

results in:

JOIN_Constraints_Schema,test_simple_joins,0
JOIN_Constraints_Schema,test_constraint_joins_setop_dis_oby_lmt,0
GBY_Schema1,test_Groupby_Rollup,0
GBY_Schema1,test_Groupby_GroupingSets,0
GBY_Schema1,test_Groupby_Cube,0
GBY_Schema1,test_gby,0
3

You may want to try a single awk approach like

awk -F: '/internal opt/ {print L4, L5, $NF} {gsub (/[ -]+/, ""); L4 = $4; L5 = $5}' OFS=, file
JOIN_Constraints_Schema,test_simple_joins, 0
JOIN_Constraints_Schema,test_constraint_joins_setop_dis_oby_lmt, 0
GBY_Schema1,test_Groupby_Rollup, 0
GBY_Schema1,test_Groupby_GroupingSets, 0
GBY_Schema1,test_Groupby_Cube, 0
GBY_Schema1,test_gby, 0