My for loop decides to become an infinite loop?

Shell Programming and Scripting
1

Hi, I was debating if I should put this in the dummies or scripts section, I apologize in advance if I chose poorly.

Fairly new to Unix and BASH scripting but I thought I made it fairly well given my limited understanding. However, the output indicates that it's looping and I'm ending up with a stupid large number of files.

If you fine folks here could help me puzzle out a more efficient code, I would greatly appreciate it.

for dir in `ls -d1 /raid/r04/wconnor/Sites/*_30km`; do
#Find the list of directories my files are in, there are several 

    for res in modqkm mod375m modhkm mod750m mod1km; do
    #This is important to the ./spatbsq program 

        for file in `ls -S1 $dir | grep -v .hdr`; do
        #Each directory contains a list of files, [file] and [file].hdr. this is to just grab all the non-.hdr files

        ./spatbsq -DS -F-9999 -B1000x6x1000 hyp2${res}.par ${dir}/${file} ${dir}/${file}.${res}
        cp ${dir}/${file}.hdr ${dir}/${file}.${res}.hdr
        done
        #runs the spatbsq on the list above and then spits out the same filename with the ${res} on the end for both [file] and [file].hdr
    done
done

He's the output from a folder with only one [file] and associated hdr. I don't know how to get the forum to make a scroll window so sorry for the long post. I've sorted it by date to try and help me understand what the script is doing. I'm pretty sure the problem arises because of the "for file in `ls -S1 $dir | grep -v .hdr`" but I'm at a loss of how I could rewrite this so it would only read it once. I've had no luck trying to set it as a variable but that may just be because I don't know how to do it properly. Perhaps outputting the ls command to a file then doing a cat as the argument in the for loop?

total 375940
-rw-rw-r-- 1 wconnor proj      852 Dec  5 13:44 lndsr_L71116034_03420090406_GDK_30km.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  5 13:44 lndsr_L71116034_03420090406_GDK_30km
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:19 lndsr_L71116034_03420090406_GDK_30km.modqkm
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:19 lndsr_L71116034_03420090406_GDK_30km.modqkm.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:20 lndsr_L71116034_03420090406_GDK_30km.mod375m
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:20 lndsr_L71116034_03420090406_GDK_30km.mod375m.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:21 lndsr_L71116034_03420090406_GDK_30km.modqkm.mod375m
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:21 lndsr_L71116034_03420090406_GDK_30km.modqkm.mod375m.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:22 lndsr_L71116034_03420090406_GDK_30km.modhkm
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:22 lndsr_L71116034_03420090406_GDK_30km.modhkm.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:24 lndsr_L71116034_03420090406_GDK_30km.mod375m.modhkm
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:24 lndsr_L71116034_03420090406_GDK_30km.mod375m.modhkm.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:25 lndsr_L71116034_03420090406_GDK_30km.modqkm.modhkm
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:25 lndsr_L71116034_03420090406_GDK_30km.modqkm.modhkm.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:26 lndsr_L71116034_03420090406_GDK_30km.modqkm.mod375m.modhkm
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:26 lndsr_L71116034_03420090406_GDK_30km.modqkm.mod375m.modhkm.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:30 lndsr_L71116034_03420090406_GDK_30km.mod750m
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:30 lndsr_L71116034_03420090406_GDK_30km.mod750m.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:32 lndsr_L71116034_03420090406_GDK_30km.mod375m.mod750m
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:32 lndsr_L71116034_03420090406_GDK_30km.mod375m.mod750m.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:34 lndsr_L71116034_03420090406_GDK_30km.mod375m.modhkm.mod750m
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:34 lndsr_L71116034_03420090406_GDK_30km.mod375m.modhkm.mod750m.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:36 lndsr_L71116034_03420090406_GDK_30km.modhkm.mod750m
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:36 lndsr_L71116034_03420090406_GDK_30km.modhkm.mod750m.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:39 lndsr_L71116034_03420090406_GDK_30km.modqkm.mod750m
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:39 lndsr_L71116034_03420090406_GDK_30km.modqkm.mod750m.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:41 lndsr_L71116034_03420090406_GDK_30km.modqkm.mod375m.mod750m
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:41 lndsr_L71116034_03420090406_GDK_30km.modqkm.mod375m.mod750m.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:43 lndsr_L71116034_03420090406_GDK_30km.modqkm.mod375m.modhkm.mod750m
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:43 lndsr_L71116034_03420090406_GDK_30km.modqkm.mod375m.modhkm.mod750m.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:44 lndsr_L71116034_03420090406_GDK_30km.modqkm.modhkm.mod750m
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:44 lndsr_L71116034_03420090406_GDK_30km.modqkm.modhkm.mod750m.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:51 lndsr_L71116034_03420090406_GDK_30km.mod1km
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:51 lndsr_L71116034_03420090406_GDK_30km.mod1km.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:56 lndsr_L71116034_03420090406_GDK_30km.mod375m.mod1km
-rw-rw-r-- 1 wconnor proj      852 Dec  6 17:56 lndsr_L71116034_03420090406_GDK_30km.mod375m.mod1km.hdr
-rw-rw-r-- 1 wconnor proj 12000000 Dec  6 17:59 lndsr_L71116034_03420090406_GDK_30km.mod375m.mod750m.mod1km

..  It goes on further but you get the idea
2

That's a dangerous use of backticks, useless use of ls *, and useless use of cat. You can't cram arbitrary sizes of things into backticks and shouldn't try. Either use pipes or temp files.

I can't quite understand what you want it to be doing from your malfunctioning program. Could you show what files it's supposed to be using, what files it's supposed to get out, and which files create what?

I'd start with:

# Find all files inside /raid/r04/wconnor/Sites/*_30km that aren't named '*.hdr', save to temp file.
# The temp file won't change later, so 'find' can't feed your own files created files back into your script.
find /raid/r04/wconnor/Sites/*_30km -type f '!' -name '*.hdr' > /tmp/$$

# Read lines from temp file
while read FILE
do
        ...
done < /tmp/$$

rm -f /tmp/$$

...but what you want done with the files isn't too clear.

1 Like
3

@Corona

This is exactly what I was hoping for. You're telling me that how I think about scripting is garbage and why it's bad. Although telling me how poorly I've written my script, I feel that this better belongs in the noob section.

What I'm trying to do is a bit obtuse so please forgive me if it's poorly explained:
I have a series of satellite images from various locations which are named along the lines of:
Image_date_location
Image_date_location.hdr

These images are of a pixel size (30m) but I'm going to fuzz them out to simulate larger pixel footprints, (250m, 375m, 500m, and 1km); this is accomplished with spatbsq and specifying the parameter $res.

After the 'spatbsq' is done, it should output:
Image_date_location.250m
Image_date_location.250m.hdr
and so on for all I would in theory end up with 4 new files for every original file, one for each resolution.

It's good to know that I shouldn't use backticks in these cases
Could you (or someone else) possibly explain to me what 'done < /tmp/$$' does and why $$ is used instead of a file name?

4

$$ is just the PID of the shell, a decimal number. It's used as a unique temporary file name. Getting it from mktemp is more technically correct but some people don't have that.

Thanks for the explanation, but I'm still missing a lot of pieces. These 'par' files you seem to be using with every file to create different files for instance. Is running the utility x times for one set of files what's supposed to be happening, each time with a different .par file?

And what's the cp for? Why copy the same file into a new name over and over? Is this ./spatbsq command altering the original files each time you run it?

How about a hypothetical example. You have only one set of files, file and file.hdr. What are the exact commands you run to convert them?

1 Like
5

Thanks again for the response, it's greatly appreciated. My .par files are located in the same directory as sparbsq and the script. Without getting into the details of what they contain, there are 5 of them listed as:
hyp2modqkm.par, hyp2mod375m.par, hyp2modhkm.par, hyp2mod750m.par, hyp2mod1km.par

I need to run spatbsq on every image file 5 times, each time with a different .par as an input.

Hope hypothetical example v2.0 will be more clear about what I need to do. My files are:
/newyork/brooklynbridge
/newyork/brooklynbridge.hdr

I would run it roughly as follows:
./spatbsq [.par] [input] [output]
./spatbsq hyp2qkm.par /newyork/brooklynbridge /newyork/brooklynbridge.qkm
./spatbsq hyp2375m.par /newyork/brooklynbridge /newyork/brooklynbridge.375m
./spatbsq hyp2hkm.par /newyork/brooklynbridge /newyork/brooklynbridge.hkm
./spatbsq hyp2750m.par /newyork/brooklynbridge /newyork/brooklynbridge.750m
./spatbsq hyp21km.par /newyork/brooklynbridge /newyork/brooklynbridge.1km

so in regards to renaming, all I do is essentially tack on the suffix of the .par to my output file. The cp is because each of these outputs need a .hdr file of the same name. The .hdr is simply spatial information for the photo and since the brooklyn bridge doesn't really shift around, I just copy the original .hdr to one with the appropriate $res suffix.

---------- Post updated at 02:06 PM ---------- Previous update was at 01:29 PM ----------

It looks like your little script snippit solved all my problems plus I feel like I've learned a lot as well, for which I'm greatful.

One quick thing though. When you have the "while read FILE", how does it know that the FILE is the /tmp/$$ file?

6

Glad it worked for you.

It doesn't -- it just reads from standard input. But the </tmp/$$ tells it what "standard input" is supposed to be, at the bottom of the loop.

Imagine it like this:

read FILE </tmp/$$

...except that there's an entire loop involved:

while read FILE
do
...
done </tmp/$$

...in which the file gets opened once and put into standard input for read and anything else inside the loop to read as it pleases.

You can do that, in shell -- redirect the input and/or output of entire loops and statements:

# all 5 echo statements go into the same file
for N in 1 2 3 4 5
do
        echo $N
done > output