Need help parsing data with sed and/or nawk

BRH · March 5, 2012, 10:55am

Good day all.

I have the following entries of data in a file in a column, however, I need this data written on a single line with several parameters in a different order.

Current format:

Treatment
      ,parmeter1=value
      ,parmeter2=value
      ,parmeter3=value
      ,parmeter4=value
      ,parmeter5=value
      ,parmeter6=value
      ,parmeter7=value
      ,parmeter8=value
      ,parmeter9=value
      ,parmeter10=value
      ,parmeter11=value
      ,parmeter12=value
      ,parmeter13=value
      ,parmeter14=value
      ,parmeter15=value
;;
Treatment
      ,parmeter1=value
      ,parmeter2=value
      ,parmeter3=value
      ,parmeter4=value
      ,parmeter5=value
      ,parmeter6=value
      ,parmeter7=value
      ,parmeter8=value
      ,parmeter9=value
      ,parmeter10=value
      ,parmeter11=value
      ,parmeter12=value
      ,parmeter13=value
      ,parmeter14=value
      ,parmeter15=value
;;

The above should look like this:

Treatment ,parmeter1=value,parmeter2=value,parmeter5=value,parmeter3=value,parmeter4=value,parmeter6=value,parmeter7=value,parmeter10=value,parmeter9=value,parmeter11=value,,parmeter12=value,parmeter15=value,parmeter14=value,parmeter13=value;;
Treatment ,parmeter1=value,parmeter2=value,parmeter5=value,parmeter3=value,parmeter4=value,parmeter6=value,parmeter7=value,parmeter10=value,parmeter9=value,parmeter11=value,,parmeter12=value,parmeter15=value,parmeter14=value,parmeter13=value;;

Thanks in advance,

Bjoern

Corona688 · March 5, 2012, 11:00am

$ cat treatment.awk

BEGIN { FS="," }

!T { T=$1" "; next }
T && /^[ \t]*,/ { T=T "," $2 }
/;;/ { print T; T="" }

$ awk -f treatment.awk data

Treatment ,parmeter1=value,parmeter2=value,parmeter3=value,parmeter4=value,parmeter5=value,parmeter6=value,parmeter7=value,parmeter8=value,parmeter9=value,parmeter10=value,parmeter11=value,parmeter12=value,parmeter13=value,parmeter14=value,parmeter15=value
Treatment ,parmeter1=value,parmeter2=value,parmeter3=value,parmeter4=value,parmeter5=value,parmeter6=value,parmeter7=value,parmeter8=value,parmeter9=value,parmeter10=value,parmeter11=value,parmeter12=value,parmeter13=value,parmeter14=value,parmeter15=value

$

joeyg · March 5, 2012, 11:04am

$ cat sample1.txt
Treatment
      ,parmeter1=value
      ,parmeter2=value
      ,parmeter3=value
      ,parmeter4=value
      ,parmeter5=value
      ,parmeter6=value
      ,parmeter7=value
      ,parmeter8=value
      ,parmeter9=value
      ,parmeter10=value
      ,parmeter11=value
      ,parmeter12=value
      ,parmeter13=value
      ,parmeter14=value
      ,parmeter15=value
;;
Treatment
      ,parmeter1=value
      ,parmeter2=value
      ,parmeter3=value
      ,parmeter4=value
      ,parmeter5=value
      ,parmeter6=value
      ,parmeter7=value
      ,parmeter8=value
      ,parmeter9=value
      ,parmeter10=value
      ,parmeter11=value
      ,parmeter12=value
      ,parmeter13=value
      ,parmeter14=value
      ,parmeter15=value
;;

$ sed 's/;;/;;~/g' sample1.txt | tr -d "\n" | tr "~" "\n" | tr -s " "
Treatment ,parmeter1=value ,parmeter2=value ,parmeter3=value ,parmeter4=value ,parmeter5=value ,parmeter6=value ,parmeter7=value ,pa
rmeter8=value ,parmeter9=value ,parmeter10=value ,parmeter11=value ,parmeter12=value ,parmeter13=value ,parmeter14=value ,parmeter15
=value;;
Treatment ,parmeter1=value ,parmeter2=value ,parmeter3=value ,parmeter4=value ,parmeter5=value ,parmeter6=value ,parmeter7=value ,pa
rmeter8=value ,parmeter9=value ,parmeter10=value ,parmeter11=value ,parmeter12=value ,parmeter13=value ,parmeter14=value ,parmeter15
=value;;

I put a ~ character at end, so when I get rid of new-lines, I can put a Return back in the data. I also suppress extra blank space characters.

BRH · March 5, 2012, 1:28pm

joeyg:

$ cat sample1.txt
Treatment
   ,parmeter1=value
   ,parmeter2=value
   ,parmeter3=value
   ,parmeter4=value
   ,parmeter5=value
   ,parmeter6=value
   ,parmeter7=value
   ,parmeter8=value
   ,parmeter9=value
   ,parmeter10=value
   ,parmeter11=value
   ,parmeter12=value
   ,parmeter13=value
   ,parmeter14=value
   ,parmeter15=value
;;
Treatment
   ,parmeter1=value
   ,parmeter2=value
   ,parmeter3=value
   ,parmeter4=value
   ,parmeter5=value
   ,parmeter6=value
   ,parmeter7=value
   ,parmeter8=value
   ,parmeter9=value
   ,parmeter10=value
   ,parmeter11=value
   ,parmeter12=value
   ,parmeter13=value
   ,parmeter14=value
   ,parmeter15=value
;;

$ sed 's/;;/;;~/g' sample1.txt | tr -d "\n" | tr "~" "\n" | tr -s " "
Treatment ,parmeter1=value ,parmeter2=value ,parmeter3=value ,parmeter4=value ,parmeter5=value ,parmeter6=value ,parmeter7=value ,pa
rmeter8=value ,parmeter9=value ,parmeter10=value ,parmeter11=value ,parmeter12=value ,parmeter13=value ,parmeter14=value ,parmeter15
=value;;
Treatment ,parmeter1=value ,parmeter2=value ,parmeter3=value ,parmeter4=value ,parmeter5=value ,parmeter6=value ,parmeter7=value ,pa
rmeter8=value ,parmeter9=value ,parmeter10=value ,parmeter11=value ,parmeter12=value ,parmeter13=value ,parmeter14=value ,parmeter15
=value;;

I put a ~ character at end, so when I get rid of new-lines, I can put a Return back in the data. I also suppress extra blank space characters.

corona688:

$ cat treatment.awk

BEGIN { FS="," }

!T { T=$1" "; next }
T && /^[ \t]*,/ { T=T "," $2 }
/;;/ { print T; T="" }

$ awk -f treatment.awk data

Treatment ,parmeter1=value,parmeter2=value,parmeter3=value,parmeter4=value,parmeter5=value,parmeter6=value,parmeter7=value,parmeter8=value,parmeter9=value,parmeter10=value,parmeter11=value,parmeter12=value,parmeter13=value,parmeter14=value,parmeter15=value
Treatment ,parmeter1=value,parmeter2=value,parmeter3=value,parmeter4=value,parmeter5=value,parmeter6=value,parmeter7=value,parmeter8=value,parmeter9=value,parmeter10=value,parmeter11=value,parmeter12=value,parmeter13=value,parmeter14=value,parmeter15=value

$

Thanks for the replies, Joey, Corona. However, you missed the last part of the challenge: the order of the parameters has to change as well.

Regards,
Bjoern

joeyg · March 5, 2012, 2:26pm

So, is the order?
1,2,5,3,4,6,7,10,9,11,?,12,15,14,13
where ? is 8 suppressed

BRH · March 5, 2012, 3:38pm

I guess I forgot the 8, this being an example. Also I'm not sure of the order yet, but if I can get an idea as to how to stream the order to a single line, I'd be very grateful.

Thanks,
Bjoern

Corona688 · March 6, 2012, 11:36am

How to stream the order really depends on what defines the order. Is parameter13 given last because it's called parameter13, or because it happened to be the 13th in the order and you always want the 13th parameter there?

---------- Post updated at 10:36 AM ---------- Previous update was at 10:22 AM ----------

If it's the order and not their names:

$ cat treatment2.awk

BEGIN { FS=","; OFS=","
        split("1,2,5,3,4,6,7,10,9,11,10,12,15,14,13", ORDER);
}

!T { TITLE=$1 ; T=1 ; next }

/^[ \t]*,/ { C[T++]=$2 }

/;;/ {  $0="";  T=""
        for(X in ORDER) $(ORDER[X])=" "C[X];
        $1=TITLE $1
        print
}

$ awk -f treatment2.awk data

Treatment parmeter1=value, parmeter2=value, parmeter4=value, parmeter5=value, parmeter3=value, parmeter6=value, parmeter7=value,, parmeter9=value, parmeter11=value, parmeter10=value, parmeter12=value, parmeter15=value, parmeter14=value, parmeter13=value
Treatment parmeter1=value, parmeter2=value, parmeter4=value, parmeter5=value, parmeter3=value, parmeter6=value, parmeter7=value,, parmeter9=value, parmeter11=value, parmeter10=value, parmeter12=value, parmeter15=value, parmeter14=value, parmeter13=value

$

Corona688 · March 6, 2012, 12:06pm

Or if it's defined by names:

$ cat treatment3.awk

BEGIN { FS=","; OFS=","
        split("parmeter1,parmeter2,parmeter5,parmeter3,parmeter4,parmeter6,parmeter7,parmeter10,parmeter9,parmeter11,,parmeter12,parmeter15,parmeter14,parmeter13", ORDER);
}

!T { TITLE=$1 ; T=1 ; next }

/^[ \t]*,/ {    split($2, Z, "=");
                VALUE[Z[1]]=Z[2];       }

/;;/ {  $0="";  T=""
        for(X in ORDER)
        {
                if(ORDER[X]) $X=" "ORDER[X]"="VALUE[ORDER[X]];
                else    $X=""
        }

        $1=TITLE$1
        $NF= $NF ";;"
        print
}

$ awk -f treatment3.awk data

Treatment parmeter1=value, parmeter2=value, parmeter5=value, parmeter3=value, parmeter4=value, parmeter6=value, parmeter7=value, parmeter10=value, parmeter9=value, parmeter11=value,, parmeter12=value, parmeter15=value, parmeter14=value, parmeter13=value;;
Treatment parmeter1=value, parmeter2=value, parmeter5=value, parmeter3=value, parmeter4=value, parmeter6=value, parmeter7=value, parmeter10=value, parmeter9=value, parmeter11=value,, parmeter12=value, parmeter15=value, parmeter14=value, parmeter13=value;;

$