Needed shell script to read txt file and do some modification

Akshay_Hegde · November 21, 2012, 3:27am

Hi ...programmers...

I need a shell script to perform some specific task..

my txt file looks like this

netcdf new {
dimensions:
    XAX1_11 = 11 ;
variables:
    double XAX1_11(XAX1_11) ;
        XAX1_11:point_spacing = "even" ;
        XAX1_11:axis = "X" ;
    float DEPTH(XAX1_11) ;
        DEPTH:missing_value = -1.e+34f ;
        DEPTH:_FillValue = -1.e+34f ;
        DEPTH:long_name = "DEPTH" ;
        DEPTH:history = "From new.txt" ;
    float TEMPERATURE(XAX1_11) ;
        TEMPERATURE:missing_value = -1.e+34f ;
        TEMPERATURE:_FillValue = -1.e+34f ;
        TEMPERATURE:long_name = "TEMPERATURE" ;
        TEMPERATURE:history = "From new.txt" ;
    float SALINITY(XAX1_11) ;
        SALINITY:missing_value = -1.e+34f ;
        SALINITY:_FillValue = -1.e+34f ;
        SALINITY:long_name = "SALINITY" ;
        SALINITY:history = "From new.txt" ;

// global attributes:
        :history = "FERRET V6.82   21-Nov-12" ;
        :Conventions = "CF-1.0" ;
data:

 XAX1_11 = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 ;

 DEPTH = 1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50 ;

 TEMPERATURE = 28.25, 28.26, 28.23, 28.22, 28.2, 28.19, 28.18, 28.18, 28.18, 
    28.19, 28.19 ;

 SALINITY = 36.42, 36.42, 36.43, 36.43, 36.42, 36.43, 36.43, 36.43, 36.44, 
    36.45, 36.44 ;
}

I have of need of output in following way

DEPTH    TEMPR    SALINITY
1    28.25    36.42
5    28.26    36.42
10    28.23    36.43
15    28.22    36.43
20    28.2    36.42
25    28.19    36.43
30    28.18    36.43
35    28.18    36.43
40    28.18    36.44
45    28.19    36.45
50    28.19    36.44

kindly programmers give me the script to get this output, I have lot of txt files of same format..

Thanks in advance

Akshay

pamu · November 21, 2012, 3:38am

Please use code tags for code and data sample

try

awk -F "[=;]" '/DEPTH/{n=split($2,DP,",")}
/TEMPERATURE/{m=split($2,TM,",")}
/SALINITY/{o=split($2,SL,",")}END{print "DEPTH","TEMPERATURE","SALINITY";
for(i=1;i<=m;i++){print DP,TM,SL}}' file

DEPTH TEMPERATURE SALINITY
 1  28.25  36.42
 5  28.26  36.42
 10  28.23  36.43
 15  28.22  36.43
 20  28.2  36.42
 25  28.19  36.43
 30  28.18  36.43
 35  28.18  36.43
 40  28.18  36.44
 45 28.19 36.45
 50   28.19   36.44

Akshay_Hegde · November 22, 2012, 5:57am

commands not working, and unable to read .txt file,
is there any other solution ?

pamu · November 22, 2012, 5:59am

What you have tried? Please show us.

Akshay_Hegde · November 22, 2012, 6:00am

Dear Pamu your solution not working...I had pasted as you given in my script...

---------- Post updated at 06:00 AM ---------- Previous update was at 05:59 AM ----------

for file in *.txt; do
echo $file
awk -F "[=;]" '/DEPTH/{n=split($2,DP,",")}
/TEMPERATURE/{m=split($2,TM,",")}
/SALINITY/{o=split($2,SL,",")}END{print "DEPTH","TEMPERATURE","SALINITY";
for(i=1;i<=m;i++){print DP,TM,SL}}' file

done

pamu · November 22, 2012, 6:03am

It should be $file .

$ cat F1.txt
netcdf new {
dimensions:
    XAX1_11 = 11 ;
variables:
    double XAX1_11(XAX1_11) ;
        XAX1_11:point_spacing = "even" ;
        XAX1_11:axis = "X" ;
    float DEPTH(XAX1_11) ;
        DEPTH:missing_value = -1.e+34f ;
        DEPTH:_FillValue = -1.e+34f ;
        DEPTH:long_name = "DEPTH" ;
        DEPTH:history = "From new.txt" ;
    float TEMPERATURE(XAX1_11) ;
        TEMPERATURE:missing_value = -1.e+34f ;
        TEMPERATURE:_FillValue = -1.e+34f ;
        TEMPERATURE:long_name = "TEMPERATURE" ;
        TEMPERATURE:history = "From new.txt" ;
    float SALINITY(XAX1_11) ;
        SALINITY:missing_value = -1.e+34f ;
        SALINITY:_FillValue = -1.e+34f ;
        SALINITY:long_name = "SALINITY" ;
        SALINITY:history = "From new.txt" ;

// global attributes:
        :history = "FERRET V6.82   21-Nov-12" ;
        :Conventions = "CF-1.0" ;
data:

 XAX1_11 = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11 ;

 DEPTH = 1, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50 ;

 TEMPERATURE = 28.25, 28.26, 28.23, 28.22, 28.2, 28.19, 28.18, 28.18, 28.18,28.19, 28.19 ;

 SALINITY = 36.42, 36.42, 36.43, 36.43, 36.42, 36.43, 36.43, 36.43, 36.44,36.45, 36.44 ;
}

$ awk -F "[=;]" '/DEPTH/{n=split($2,DP,",")}
/TEMPERATURE/{m=split($2,TM,",")}
/SALINITY/{o=split($2,SL,",")}END{print "DEPTH","TEMPERATURE","SALINITY";
for(i=1;i<=m;i++){print DP,TM,SL}}' F1.txt
DEPTH TEMPERATURE SALINITY
 1  28.25  36.42
 5  28.26  36.42
 10  28.23  36.43
 15  28.22  36.43
 20  28.2  36.42
 25  28.19  36.43
 30  28.18  36.43
 35  28.18  36.43
 40  28.18  36.44
 45 28.19 36.45
 50   28.19   36.44

Akshay_Hegde · November 22, 2012, 6:34am

I had pasted as you given not even modified single word..pls help

and I need output in column wise depth in one column, temperature in one column,,,,so on....

pamu · November 22, 2012, 6:41am

Please use code tags for code and data sample.

Try.. this as per what your input i see from here..:rolleyes:

awk -F "[=;]" '/DEPTH/{n=split($2,DP,",")}
/TEMPERATURE/{s=$2;getline;s=s""$1;m=split(s,TM,",")}
/SALINITY/{s=$2;getline;s=s""$1;split(s,SL,",")}END{print "DEPTH","TEMPERATURE","SALINITY";
for(i=1;i<=m;i++){print DP,TM,SL}}' F1.txt

Akshay_Hegde · November 22, 2012, 7:05am

see pamu this is output but all values not coming...some are missing

akshay@akshay-Vostro-460:~/NCL/Untitled Folder$ ./new.sh
DEPTH TEMPERATURE SALINITY
 1  28.25  36.42
 5  28.26  36.42
 10  28.23  36.43
 15  28.22  36.43
 20  28.2  36.42
 25  28.19  36.43
 30  28.18  36.43
 35  28.18  36.43
 40  28.18  36.44
 45    
akshay@akshay-Vostro-460:~/NCL/Untitled Folder$

pamu · November 22, 2012, 7:12am

Have you tried my code(post 8)...?

Akshay_Hegde · November 22, 2012, 7:18am

Yes its displaying output...whether it is possible to write data into new file..and every time its difficult to edit script because I have files (of type F1.txt )about 290

pamu · November 22, 2012, 7:23am

Yes we can write data into new file

if you want to create separate file
use

for file in *.txt; do
awk -F "[=;]" '/DEPTH/{n=split($2,DP,",")}
/TEMPERATURE/{s=$2;getline;s=s""$1;m=split(s,TM,",")}
/SALINITY/{s=$2;getline;s=s""$1;split(s,SL,",")}END{print "DEPTH","TEMPERATURE","SALINITY";
for(i=1;i<=m;i++){print DP,TM,SL}}' $file > "out_"$file
done

Or if you want to write into a one single file
use

for file in *.txt; do
awk -F "[=;]" '/DEPTH/{n=split($2,DP,",")}
/TEMPERATURE/{s=$2;getline;s=s""$1;m=split(s,TM,",")}
/SALINITY/{s=$2;getline;s=s""$1;split(s,SL,",")}END{print "DEPTH","TEMPERATURE","SALINITY";
for(i=1;i<=m;i++){print DP,TM,SL}}' $file >> Output_file
done

pamu

Akshay_Hegde · November 22, 2012, 7:40am

Hats off to you...Great Solution ! I wasn't expected that it will be solved..thank you so much....

---------- Post updated at 07:40 AM ---------- Previous update was at 07:34 AM ----------

Just now I had tried with one big file.....its not working

RudiC · November 22, 2012, 7:44am

Well, you might want to try this one, based on pamu's suggestion, adapted to your multiline records. May not work on all versions of awk:

awk  '{gsub (/[ ;{}]/,"")}
     /^DEPTH=/       {m=split($2, D, /,\n*/)}
     /^TEMPERATURE=/ {n=split($2, T, /,\n*/)}
     /^SALINITY=/    {o=split($2, S, /,\n*/)}
     END {for (i=1;i<=n; i++) print D,T,S}
    ' FS="=" OFS="\t" RS= file
1     28.25    36.42
5     28.26    36.42
10    28.23    36.43
15    28.22    36.43
20    28.2     36.42
25    28.19    36.43
30    28.18    36.43
35    28.18    36.43
40    28.18    36.44
45    28.19    36.45
50    28.19    36.44

Akshay_Hegde · November 22, 2012, 8:06am

[LEFT]Dear RudiC..
I had tried with big file in which depth is coming at starting of declaration....might be because of conflict...I am here pasting the same
netcdf \175_phy {
dimensions:
XLON = 6 ;
YLAT = 7 ;
DEPTH = 462 ;
bnds = 2 ;
XIN = 3968 ;
variables:
double XLON(XLON) ;
XLON:units = "degrees_east" ;
XLON:point_spacing = "even" ;
XLON:axis = "X" ;
XLON:modulo = 360. ;
XLON:standard_name = "longitude" ;
double YLAT(YLAT) ;
YLAT:units = "degrees_north" ;
YLAT:point_spacing = "even" ;
YLAT:axis = "Y" ;
YLAT:standard_name = "latitude" ;
double ZDEPTH(ZDEPTH) ;
LON:history = "From sk175_ctd.csv" ;
double LAT(XIN) ;
LAT:missing_value = -1.e+34 ;
LAT:_FillValue = -1.e+34 ;
LAT:long_name = "V10" ;
LAT:history = "From sk175_ctd.csv" ;

// global attributes:
:history = "FERRET V6.82 21-Nov-12" ;
:Conventions = "CF-1.0" ;
:title = "748_Physical_Data" ;
data:

XLON = 93.4369964599609, 94.4369964599609, 95.4369964599609,
96.4369964599609, 97.4369964599609, 98.4369964599609 ;

YLAT = 12.0030002594, 13.0030002594, 14.0030002594, 15.0030002594,
16.0030002594, 17.0030002594, 18.0030002594 ;

DEPTH = 2, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80,
85, 90, 95, 100, 105, 110, 115, 120, 125, 130, 135, 140, 145, 150, 155,
160, 165, 170, 175, 180, 185, 190, 195, 200, 205, 210, 215, 220, 225,
230, 235, 240, 245, 250, 255, 260, 265, 270, 275, 280, 285, 290, 295,
300, 305, 310, 315, 320, 325, 330, 335, 340, 345, 350, 355, 360, 365,
370, 375, 380, 385, 390, 395, 400, 405, 410, 415, 420, 425, 430, 435 ;
TEMP_G =
30.3838383066439, 30.3882053246992, 30.3924360699913, 30.3965465565034,
30.4005514621873, 30.404463453727,
30.374986188776, 30.3796235626175, 30.3841268994998, 30.3885127368018,
30.3927962980837, 30.3969907826882,
30.3656463921024, 30.3705491637335, 30.3753205195448, 30.3799775103733,
30.3845358989906, 30.3890094149276,
30.35582892106, 30.3609916018094, 30.3660258665825, 30.3709492694464,
30.3757781056927, 30.3805266323537,
30.3455445472965, 30.3509611316731, 30.3562526706283, 30.3614372109447,
30.366531573443, 30.371550539739 ;
}

How do I bypass

[/LEFT]

RudiC · November 22, 2012, 8:19am

Please use code tags!

The new sample file you posted is very different to the one we, pamu and myself, were working on. This ruins the solutions: no empty line between records, no TEMPERATURE entry, no SALINITY entry.
How do you expect solutions to work if the data structures they rely on disappear?

Akshay_Hegde · November 22, 2012, 8:24am

whether its possible to attach my txt.file ? here

RudiC · November 22, 2012, 8:30am

Use attachments in advanced posting.

Akshay_Hegde · November 22, 2012, 8:34am

Needed variables in column are ZDEPTH,TEMP_G, TEMP_L,SAL_G,SAL,G

see my file...here attached...

RudiC · November 22, 2012, 12:06pm

This really brings my awk down to its knees! Try this and come back with results:

awk '{gsub (/[ ;{}\n]/,"")}
     /^ZDEPTH[^_]/ {j=split($2, ZD, /, */); HdZ=$1; if (j>max) max=j}
     /^TEMP_G/     {k=split($2, TG, /, */); HdT=$1; if (k>max) max=k}
     /^TEMP_L/     {l=split($2, TL, /, */); HdL=$1; if (l>max) max=l}
     /^SAL_G/      {m=split($2, SG, /, */); HdG=$1; if (m>max) max=m}
     /^SAL_L/      {n=split($2, SL, /, */); HdS=$1; if (n>max) max=n}
     END {print HdZ, HdT, HdL, HdG, HdS; 
          for (i=1;i<=max; i++) print ZD, TG, TL, SG, SL}
    ' FS="[=;]" OFS="\t" RS= file
ZDEPTH  TEMP_G  TEMP_L  SAL_G   SAL_L
2       30.3838383066439        29.9991184776014        32.6905910485831        33.0539176423929
5       30.3882053246992        29.77256752473  32.6878127401931        33.0282461530583
10      30.3924360699913        29.9804910263879        32.685054943924 32.7374162316174
15      30.3965465565034        29.9589844353825        32.6823094722375        32.8266457559109