Getting information from various files

Shell Programming and Scripting
1

I have got this piece of csh code that looks into various log files and outputs some parameters

For example, I might have 4 files and want to grep for the lines containing "Best Value"

npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log

Doing

grep "Best Value" *dc0p004-16x12drw*.log

gives

npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:0. Best Value   = 0.0348089
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:1. Best Value   = 0.0348089
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:2. Best Value   = 0.0347595
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:3. Best Value   = 0.0347595
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:4. Best Value   = 0.0343211
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:5. Best Value   = 0.0343211
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:6. Best Value   = 0.034262
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:7. Best Value   = 0.034262
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:8. Best Value   = 0.034262
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:9. Best Value   = 0.0341526
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:10. Best Value   = 0.0341407
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:11. Best Value   = 0.0341101
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:12. Best Value   = 0.0340129
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:13. Best Value   = 0.0340129
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:14. Best Value   = 0.0339073
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log:15. Best Value   = 0.0339073
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:0. Best Value   = 0.0339073
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:1. Best Value   = 0.0338745
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:2. Best Value   = 0.0338223
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:3. Best Value   = 0.0338054
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:4. Best Value   = 0.033714
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:5. Best Value   = 0.033714
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:6. Best Value   = 0.033714
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:7. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:8. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:9. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:10. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:11. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:12. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:13. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log:14. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:0. Best Value   = 0.0336161
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:1. Best Value   = 0.0336147
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:2. Best Value   = 0.0334466
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:3. Best Value   = 0.0334466
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:4. Best Value   = 0.0334466
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:5. Best Value   = 0.0334184
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:6. Best Value   = 0.0333759
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:7. Best Value   = 0.0333759
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log:8. Best Value   = 0.0333759
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:0. Best Value   = 0.0333759
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:1. Best Value   = 0.0333759
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:2. Best Value   = 0.0333759
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:3. Best Value   = 0.0333041
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:4. Best Value   = 0.0333041
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:5. Best Value   = 0.0333041
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:6. Best Value   = 0.0332733
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:7. Best Value   = 0.0332733
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:8. Best Value   = 0.0332733
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:9. Best Value   = 0.033229
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:10. Best Value   = 0.033229
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:11. Best Value   = 0.033229
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:12. Best Value   = 0.033229
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log:13. Best Value   = 0.033229

Now I want to have something like this stored in a file

0. Best Value = 0.0348089 
1. Best Value = 0.0348089
2. Best Value = 0.0347595
3. Best Value = 0.0347595
4. Best Value = 0.0343211
5. Best Value = 0.0343211
6. Best Value = 0.034262
7. Best Value = 0.034262
8. Best Value = 0.034262
9. Best Value = 0.0341526
10. Best Value = 0.0341407
11. Best Value = 0.0341101
12. Best Value = 0.0340129
13. Best Value = 0.0340129
14. Best Value = 0.0339073
15. Best Value = 0.0339073
0. Best Value = 0.0339073
1. Best Value = 0.0338745
2. Best Value = 0.0338223
3. Best Value = 0.0338054
4. Best Value = 0.033714
5. Best Value = 0.033714
6. Best Value = 0.033714
7. Best Value = 0.0336161
8. Best Value = 0.0336161
9. Best Value = 0.0336161
10. Best Value = 0.0336161
11. Best Value = 0.0336161
12. Best Value = 0.0336161
13. Best Value = 0.0336161
14. Best Value = 0.0336161
0. Best Value = 0.0336161
1. Best Value = 0.0336147
2. Best Value = 0.0334466
3. Best Value = 0.0334466
4. Best Value = 0.0334466
5. Best Value = 0.0334184
6. Best Value = 0.0333759
7. Best Value = 0.0333759
8. Best Value = 0.0333759
0. Best Value = 0.0333759
1. Best Value = 0.0333759
2. Best Value = 0.0333759
3. Best Value = 0.0333041
4. Best Value = 0.0333041
5. Best Value = 0.0333041
6. Best Value = 0.0332733
7. Best Value = 0.0332733
8. Best Value = 0.0332733
9. Best Value = 0.033229
10. Best Value = 0.033229
11. Best Value = 0.033229
12. Best Value = 0.033229
13. Best Value = 0.033229

As one may notice the 0. value occurs multiple times. I need to remove the entry 0. except on the first occurance which is in the first line

The code below actually does that.

However if I have only one file, the script does not work. Also I want to change the file naming system to

npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run1.log
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run2.log
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run3.log
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw-run4.log

which means that if the log files are a continuation of runs, the first run is tagged with -run1- as well. Of course, if only one run is done the lof file will still be
npt02-z30-sr65-rgdt0p50-dc0p004-16x12drw.log, i.e. without the -run1- tag.

I would also need to run the script on such a file and create a file with the parameters of interest.

The csh code is shown below, but I am getting confused how to change it so that I will pass multiple files all having a -run- tag. The code below ignored the -run- tag as currently the first run never has the -run1- tag.

    foreach f ($fnames)

      echo "(I) ${f}.log, (O) ${f}.mis"

      set fdrwtag = `echo $f | awk '/drw/'`
      set fsmptag = `echo $f | awk '/smp/'`

      if ($fdrwtag != "") then

        grep "Rms Value" $f*.log              \
          | awk '{ if ($0 !~ /run[0-9]+\.log:0\. /)  \
                   { sub(/[^:]*:[0-9]*\.[ \t]*/,x); print i++". " $0 }  \
                 }' > $f.mis

        grep "Best Value" $f*.log             \
          | awk '{ if ($0 !~ /run[0-9]+\.log:0\. /)  \
                   { sub(/[^:]*:[0-9]*\.[ \t]*/,x); print i++". " $0 }  \
                 }' >> $f.mis

      else if ($fsmptag != "") then

        grep "Best Value" $f*.log             \
          | awk '{ if ($0 !~ /run[0-9]+\.log:0\. /)  \
                   { sub(/[^:]*:[0-9]*\.[ \t]*/,x); print i++". " $0 }  \
                 }' >> $f.mis

    end
2

Try this,

awk '{if(p!=FILENAME){p=FILENAME;i=0}if(/Best Value/){print i++".",$0}}' *dc0p004-16x12drw*.log
3

Usually when using grep on various files, one get the file name on each line and then the matching line.

How can I get the file name with the matching line when using just one file?

for example

grep "Rms Value" *dc0p006-16x12drw-run*.log

gives

npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:0. Rms Value    = 0.00551131
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:1. Rms Value    = 0.0105689
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:2. Rms Value    = 0.0137097
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:3. Rms Value    = 0.0163874
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:4. Rms Value    = 0.0187422
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:5. Rms Value    = 0.0197708
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:6. Rms Value    = 0.0208463
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:7. Rms Value    = 0.0217194
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:8. Rms Value    = 0.0226114
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:9. Rms Value    = 0.0229611
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:10. Rms Value    = 0.0239331
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:11. Rms Value    = 0.025327
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:12. Rms Value    = 0.0250741
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:13. Rms Value    = 0.0255705
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:14. Rms Value    = 0.0266015
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:15. Rms Value    = 0.0268534
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:16. Rms Value    = 0.0264882
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:17. Rms Value    = 0.0269775
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:0. Rms Value    = 0.0269775
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:1. Rms Value    = 0.0271278
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:2. Rms Value    = 0.0267241
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:3. Rms Value    = 0.0267395
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:4. Rms Value    = 0.0267743
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:5. Rms Value    = 0.0268856
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:6. Rms Value    = 0.0273428
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:7. Rms Value    = 0.0283749
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:8. Rms Value    = 0.0278549

but

grep "Rms Value" *dc0p006-16x12drw-run1.log

gives

0. Rms Value    = 0.00551131
1. Rms Value    = 0.0105689
2. Rms Value    = 0.0137097
3. Rms Value    = 0.0163874
4. Rms Value    = 0.0187422
5. Rms Value    = 0.0197708
6. Rms Value    = 0.0208463
7. Rms Value    = 0.0217194
8. Rms Value    = 0.0226114
9. Rms Value    = 0.0229611
10. Rms Value    = 0.0239331
11. Rms Value    = 0.025327
12. Rms Value    = 0.0250741
13. Rms Value    = 0.0255705
14. Rms Value    = 0.0266015
15. Rms Value    = 0.0268534
16. Rms Value    = 0.0264882
17. Rms Value    = 0.0269775
4

Use -H option with grep.

5 
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:0. Rms Value    = 0.00551131
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:1. Rms Value    = 0.0105689
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:2. Rms Value    = 0.0137097
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:3. Rms Value    = 0.0163874
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:4. Rms Value    = 0.0187422
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:5. Rms Value    = 0.0197708
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:6. Rms Value    = 0.0208463
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:7. Rms Value    = 0.0217194
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:8. Rms Value    = 0.0226114
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:9. Rms Value    = 0.0229611
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:10. Rms Value    = 0.0239331
6

Grep will print filename at the beginning, try it with awk :

awk '/Rms Value/' *dc0p006-16x12drw-run*.log
7

I have file like below. The lines with :0. must be removed except the first one.

npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:0. Best Value   = 0.0346886
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:1. Best Value   = 0.034645
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:2. Best Value   = 0.034645
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run1.log:3. Best Value   = 0.034645
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:0. Best Value   = 0.0338255
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:1. Best Value   = 0.0338255
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:2. Best Value   = 0.0337909
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:3. Best Value   = 0.0337666
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run2.log:4. Best Value   = 0.0335744
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run3.log:0. Best Value   = 0.0335744
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run3.log:1. Best Value   = 0.0335052
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run3.log:2. Best Value   = 0.0334884
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run3.log:3. Best Value   = 0.0334884
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run4.log:0. Best Value   = 0.0331475
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run4.log:1. Best Value   = 0.0331475
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run4.log:2. Best Value   = 0.0331475
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run5.log:0. Best Value   = 0.0331475
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run5.log:1. Best Value   = 0.0331475
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run5.log:2. Best Value   = 0.0328672
npt02-z30-sr65-rgdt0p50-dc0p006-16x12drw-run5.log:3. Best Value   = 0.0328672

I have coded in csh script like this, but am looking to see if I can improve from this to simplify.
Any suggestions welcome.

          grep -H "Best Value" $f-run*.log                    \
            | awk '{ if (NR == 1) {                           \
                          chisq = $5 / var_rgdt;              \
                          print i++". "$2" "$3" "$4" "$5      \
                      } else {                                \
                          if ($0 !~ /run[0-9]+\.log:0\. /) {  \
                              chisq = $5 / var_rgdt;          \
                              print i++". "$2" "$3" "$4" "$5  \
                          }                                   \
                      }                                       \
                    }' >> $f.csmis