Use foreach with sed in a bash script

UNIX for Beginners Q & A
1

I want to extract data from a ASCII file that looks like the one provided here (see input.txt). For this purpose I used sed commands. I want to chain the sed commands into a script that I can call with custom variables, instead of having to run it multiple times (Need to run the code for 30*24 = 720 times).

Here is the first few lines of the - the block starting 1NAME repeats for a given number of times:

AVERAGE   MODELNAME -- RUNNAME
 0  1  11121    0. 11122   24.
       -9700000         4000000   0   -241200000000   -1620000
1.00000      1000.00000  10 10   1   2   0    15.    11.     0.
    1    1  500  400
NAME
          11121      0.00     11121      1.00
   1NAME
 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00 0.0000000E+00
NAME
          11121      1.00     11121      2.00
   1NAME
 1.0000000E+00 45.0000000E+00 01.0000000E+00 115.0000000E+00 5.0000000E+00
 2.0000000E+00 66.0000000E+00 09.0000000E+00 180.0000000E+00 4.0000000E+00
 3.0000000E+00 80.0000000E+00 70.0000000E+00 130.0000000E+00 5.0000000E+00

What I want to do is grab a value from a given location (after the occurrence 1NAME) from this file. As an example, say I want to grab the value 66.0000000E+00 - that occurs at position 7 (5 from row 1 + 2 from row 2).

  1. Here is what I did. Listed here are some challenges and what i did to fix it (see lines starting with >>): 1. The format is rather strange and has multiple whitespaces. So I used sed to remove multiple spaces and replace with one space. And then using the piped output from this step, removed all blank lines. This resulted in all the data in the file arranged as one value per row.

>>

sed 's/\s\+/\n/g' <input.txt>| sed '/^$/d
  1. I then used sed command to extract data from a given row (say row 11 and row 50).
sed -n -e 11p -e 50p
  1. I tried to put all these commands as a bash script with custom row number. Since I have multiple data to extract, I used foreach fuction. The script I wrote is unfortunately giving me errors. I also want outputs from the last sed command to be piped to a single output file:
#!/bin/bash #Set input file name set FILE=$cwd/sample_or_2day foreach GRID (23729) foreach GRIDTIME(28 4100) sed 's/\s+/\n/g' $FILE | sed '/^$/d' | sed '1,36d' > temp_out rm -f $cwd/out_$GRID sed -n -e "$GRIDTIME" temp_out | tee $cwd/out_$GRID

In this code, the last sed command is only throwing out one value, even though my foreach has two inputs. I thought piping the output using tee would let the file add output from sed instead of overwrite the file itself.

  1. Also, I want to modify this code so that the foreach array is self generated using an equation. How can I use the if loop for this purpose (or another command) and how can I pass the results from the if loop into foreach? As an example: foreach (1 100 199 298) could be set as: for n=99, i = 1 + 99 (looped from 1 - 298).

Thanks for your patience. I am a nervous programmer and trying to master basics. I spent time looking at sed insturction pages, and user support forums. Any recommendations are welcome - especially with explicit instructions. Thanks!

2

Welcome to the forum.

First comment on your title: there's no foreach in bash - it's from csh . bash provides similar but not identical commands.

Now to your specification. Please form sentences in plain English, carefully, detailedly, and completely. And, make sure the written request and sample code are consistent (you want "to grab the value 66.0000000E+00", but your sed scripts deliver -9700000 and 0.0000000E+00). Describe the problem and desired result. Add your (failed?) attempts: the steps you have taken and the tools you have used.

Let me try to paraphrase what I more guessed than understood from what you said above:
We have an input file with a five line general header, and followed by multiple records, each with a two line sub header, the keyword 1NAME in an extra line, and three data lines of five floating point numbers, i.e. 15 numbers.
From every record, we need to extract one certain number pointed to by a variable or an algorithm (to be defined).

Now there's some questions to be asked:

  • Are any (sub) header data relevant and need to be taken into accont or passed back?
  • how will the pointer be passed to this script / tool / program?
  • are data from every single record to be extracted?
  • how would the named algorithm be defined?
  • what should be passed back to the caller?
1 Like