Foreach loop through subdirectories in csh

Hi

You might find it very trivial but actually don't know how to loop through all sub-directories and their child directories into a csh. bash was easier I believe but here I am, stuck with csh. So elaborately here's my problem:
Let's say I have my parent directory named C-H/ under which I have C2H3O1/, C2H4O1/...let's say under C2H3O1/ I have several subdirectories like CH2CH2OH/, CH3CH2O/, CH3CHOH/. Inside each sub-directories I have several child directories likeVAS-S0001-001-Ru_001/,VAS-S0001-002-Ru_001/,VAS-S0001-003-Ru_001/. Inside all of them reside OPT-0/, OPT-1/ inside which my job input files and job scripts reside. I wrote a script that I'd like to run inside each child directory that modifies some input files and submits the job. I would like to run my script one time under C-H that'll go through each directory and submit my job.

Here's the silly thing that I have:

#!/bin/csh
set dir=`*/*/*/*/`
foreach $dir
 /scratch/scratchdirs/sol/Scripts/QMcalc/GEN_POSCAR
end

I know I'm doing something very very wrong, I don't have a counter for the foreach but how do I set that since the number of directories might be different for different cases? Thanks a lot!

You are *really* close:

#!/bin/csh
set dirs=`*/*/*/*/`
foreach dir ($dirs)
 cd $dir
 /scratch/scratchdirs/sol/Scripts/QMcalc/GEN_POSCAR
end

If you want the whole directory tree:

set dirs=`find . -type d`

Thanks a lot guys! Still I'm having this problem:

/scratch/scratchdirs/sol/Scripts/QMcalc/GEN_POSCAR: Command not found.
/scratch/scratchdirs/sol/Scripts/QMcalc/GEN_POSCAR: Command not found.
./abc/OPT-0: No such file or directory.

P.S. I just created some directories named abc, def and some subdirectories like OPT-0,OPT-1 to test it. If I directly go to the directory and run my script it works, so I think the script is fine.
smiling dragon, your code:
#!/bin/csh set dirs=`*/*/*/*/` foreach dir ($dirs) cd $dir /scratch/scratchdirs/sol/Scripts/QMcalc/GEN_POSCAR endShould it be

cd $dir

or

cd $dirs

? I tried both but for

cd $dirs

it says too many arguements. Can you kindly explain why it's

cd $dir

? Again thanks guys, really appreciate this.

dirs is a list of all directories.
The loop variable dir picks one directory at a time.
Test it with

echo $dirs
foreach dir ($dirs)
  echo $dir
end

We all thought that your script is /scratch/scratchdirs/sol/Scripts/QMcalc/GEN_POSCAR .
Is it different from the script that manually run?

No, that's the same script :confused:

Please give the shebang (1st line) of your script!

The first line is

#!/bin/bash

Is it the problem? I tried changing that to csh and then even directly the script doesn't work. I don't know why I used bash at that time though :confused:
The echo testing works. Thanks again man, appreciate it.

---------- Post updated at 05:44 AM ---------- Previous update was at 05:28 AM ----------

When I tried to change the script to csh it says something like this -

mv: No match.
sed: can't read input: No such file or directory
sed: can't read input: No such file or directory
sed: can't read input: No such file or directory
grep: input: No such file or directory
grep: input: No such file or directory
sed: can't read input: No such file or directory
cat: input: No such file or directory
rm: cannot remove `input': No such file or directory
Illegal variable name.
mv: No match.
sed: can't read input: No such file or directory
sed: can't read input: No such file or directory
sed: can't read input: No such file or directory
grep: input: No such file or directory
grep: input: No such file or directory
sed: can't read input: No such file or directory
cat: input: No such file or directory
rm: cannot remove `input': No such file or directory
Illegal variable name.
./abc/OPT-0: No such file or directory.

I checked all the input files and they're correct. :confused:

---------- Post updated at 06:12 AM ---------- Previous update was at 05:44 AM ----------

If you want to look at my script just in case:

mv *.cell input
dos2unix -q input # > /dev/null 2> /dev/null 
sed -i '1,7d' input                   
sed -i '/%ENDBLOCK POSITIONS_FRAC/,$d' input                          
sed -i '/Ru/,$d' input               
grep C input > temp                                          
grep O input >> temp                             
sed -i '/C/,$d' input                                 
cat temp input > mols                                 
rm temp input                                               
count_C=$(grep -c C mols)
count_H=$(grep -c H mols)
count_O=$(grep -c O mols)
value="   ${count_C}     ${count_O}     ${count_H}"
sed -i 's/  H   /   /g;s/  C   /   /g;s/  O   /   /g' mols              
sed -i '1,$ s/   /  /g' mols                                                              
cp /scratch/scratchdirs/saleheen/Project-005.11.07.14.EL.44/Ru_001/VCS-S00005-001-Ru001_444R/CONTCAR .
sed -i '74,$d' CONTCAR                                 
sed -i '26,41 s/T/F/g' CONTCAR                 
cat CONTCAR mols > POSCAR                             
sed -i '74,$ s/$/   T   T   T/g' POSCAR 
rm CONTCAR mols
sed -i '6,6 s/$/   C    O    H/g' POSCAR
awk -v value="$value" 'NR==7{$0=$0" "value}1' POSCAR > Temp
mv Temp POSCAR
#qsub job-hopper-vasp

Assignments like count_C=$(grep -c C mols) do not work in csh (csh would need the set keyword before).
So the shebang #!/bin/bash is correct (of course it must exist).
Your script expects to be started in a certain directory.
So put a cd command before you invoke your script,
or change your script to do the necessary cd first!

The scripts name is GEN_POSCAR, so if I say like this-

cd /scratch/scratchdirs/saleheen/Scripts/My-scripts/GEN_POSCAR

won't it say it's not a directory? Or am I understanding something wrong here? :confused:

---------- Post updated at 07:38 AM ---------- Previous update was at 07:01 AM ----------

And I should add if I put the .cell file in the same directory with the forloop script it does the work as usual which probably means it's not going to the directories cause it shows the following error message :

./abc/OPT-0: No such file or directory.

But these directories exist there :confused:

If the scripts (short) name is "GEN_POSCAR", then it is a file in directory "/scratch/scratchdirs/saleheen/Scripts/My-scripts". The full name would be "/scratch/scratchdirs/saleheen/Scripts/My-scripts/GEN_POSCAR",

but you were told by MadeInGermany to do this:

# cd /scratch/scratchdirs/saleheen/Scripts/My-scripts
# ./GEN_POSCAR

As i sense there is some confusion about the directory concept, here it is:

Every file is placed somewhere under the "root"-hierarchy, which is denominated by "/". "/" is the topmost directory on every UNIX- (and UNIX-like) system. Now, every directory can hold two (in fact there are more, i am simplifying here) types of entries: other directories and files. Each of these other directories can also hold directories and files, and so forth. Every directory si separated by the directory "above" by a slash "/".

Therefore, "/scratch/scratchdirs/saleheen/Scripts/My-scripts" means: in the root directory "/", there is a directory "scratch". Inside "/scratch" there is another directory "scratchdirs", hence: "/scratch/scratchdirs". Inside "/scratch/scratchdirs" there is the directory "salaheen" and inside this is "Scripts", asf.. Inside the bottom directory "/scratch/scratchdirs/saleheen/Scripts/My-scripts" is a file called "GEN_POSCAR", which holds your scripts source code.

Think of it like that: you open the topmost box. Inside you find several smaller boxes (subdirectories) and some loose sheets of paper (files). Opening one of the smaller boxes you find again smaller boxes (sub-subdirectories) and other loose sheets of paper. And so on, until a box contains only loose sheets and no other boxes.

I hope this helps.

bakunin

Thanks a lot for clearing that up for me, bakunin. I was just wondering, if I do cd scratch... and then ./GEN _POSCAR thing doesn't that mean I am running the script in the script's directory? Shouldn't I try to run the script in my current directory tree?
I did what you said but still it shows the same error message. I am curious, the first line of the error message

mv: cannot stat `*.cell': No such file or directory

why does it say that? I have the .cell file in my directory tree! Thanks a lot man! Really appreciate what you are doing.

The error message is coming from inside the script and it is, because where the script is there is no such file as ".cell". Note that ".cell" is meaning all the files in the current directory with a name ending in ".cell". But what "current directory" means depends on where you were when you started the script.

This is why you should always use absolute paths (paths starting from the root dir) in scripts. For instance:

#! /bin/bash
typeset WorkPath="/path/to/some/dir"
typeset File1="${WorkPath}/filename.ext"

echo "WorkPath is: $WorkPath"
echo "Filename is: $File1"

# now, notice the usage of the variable holding the path and the other
# variable holding the filename:
cp /etc/hosts "$WorkPath"
mv "$WorkPath/ls" "$File1"

# here we do something with the file. What we do doesn't matter, just to
# show the principle
wc -c "$File1" > "$File1".tmp
mv "$File1.tmp" "$File1"

exit 0

Notice, that the full path is only named once, then used throughout the script. Think about what content each of the variables holds at which time and what the complete commands with expanded variables would look like. I suggest you adapt the script a bit and play around with it.

Having said this, you cycle through all directories and call the script for each directory once, but the directories you cycle through are nowhere mentioned in your script. You probably should do something with the directory names you cycle through, otherwise there is no sense in doing this, no?

What you do is:

./dir1: script
./dir2: script
./dir2/dir1: script
./dir2/dir2: script
...

What you probably want to do is something like:

./dir1: script ./dir1
./dir2: script ./dir2
./dir2/dir1: script ./dir2/dir1
./dir2/dir2: script ./dir2/dir2
...

Supposing that the script can do something with the directory name it gets passed. To understand how to do something with passed parameters you should read the man page of your shell (mind you, "bash" or "ksh", but DEFINITELY NOT csh! Do not even consider to use csh for scripts, the damned thing is buggy to no end!).

Here is a short introduction:

#! /bin/bash
# demonstration script for parameters
typeset localvar=""

echo "my first passed parameter is: $1"

# using variables to take parameters:
localvar="$1"

echo "you see, localvar now holds the same value: $
localvar"

exit 0

save this to a file "myparam.sh", slap on the executable-bit and issue:

./myparam.sh abc
./myparam.sh abc def
./myparam.sh "abc def"
./myparam.sh ./path/to/some/file

You might want to change your script accordingly.

I hope this helps.

bakunin

Thank you so much bakunin. If I'm understanding correctly, you're telling me to use the absolute path in my script. I actually didn't get this part of your answer, I'm so sorry, I'm just a newbie.

When I say

set dirs=`find . -type d`
foreach dir ($dirs)
cd $dir

shouldn't it go to the child directories where my input files are? Normally what I do is I go to each child directory one by one and call my script like this:

/scratch/scratchdirs/saleheen/Scripts/My-scripts/GEN_POSCAR_Ru-444

The same way I wrote in the script. When I'm setting my subdirectpry tree as the variable and calling that variable through the loop one by one, why do I have to mention directory name? And this part-

What does that mean? What my understanding is ./dir1 , going into directory1 and then run the script. why there's ./dir1 at the end too? I'm so sorry man, may be I should read something before just posting my problems. Since I'm stuck with csh (Trying to go to bash) would you be kind enough to provide some easier books on csh for beginners like me?

Thank you so much again!

Actually i am telling you to use only absoulte paths in all your scripts.

Yes, it should. I looked at your script in post #1 and there it reads:

I take it that has already been dealt with, yes?

OK. What i suggest you should do is to give the directory you would go as parameter to the script. The script itself would then go to the directory, do what it has to do, and so on. I would write your script this way (all the seds put together, not used "-i" because it is not portable):

#! /bin/bash
typeset WDir="$1"
typeset WorkFile1="${WDir}/input"
typeset WorkFile2="${WDir}/temp"
typeset WorkFile3="${WDir}/mols"

                                        # a little check up front:
if [ ! -d "$WDir" ] ; then
     echo "ERROR: $WDir is not a directory, exiting..." >&2
     exit 1
fi

mv "${WDir}*.cell" "$WorkFile1"
dos2unix -q "$WorkFile1" >/dev/null 2>/dev/null 

sed  '1,7 d                   
      /%ENDBLOCK POSITIONS_FRAC/,$ d                          
      /Ru/,$ d' "$WorkFile1" > "${WorkFile1}.tmp"
mv "${WorkFile1}.tmp" "${WorkFile1}"

grep C "$WorkFile1"   > "$WorkFile2"                                          
grep O "$WorkFile1" >> "$WorkFile2"                             

sed '/C/,$d' "${WorkFile1}" > "${WorkFile1}.tmp"
mv "${WorkFile1}.tmp" "${WorkFile1}"                               

cat "$WorkFile2" "$WorkFile1" > "$WorkFile3"                                 

rm "$WorkFile2" "$WorkFile1"
....

etc., etc.. Notice that the names of the 3 files you use are defined only once in the whole script, so it is easy to change them. Also notice, that because they are all declared with a full path the script only uses absolute paths now.

/PS: Furthermore, the script now understands the directory to work on as a parameter, so instead of the (inherently error-prone)

for dir in (...some list...) ; do
     cd $dir
     script
done

you can do the (much cleaner)

for dir in (...some list...) ; do
     script $dir
done

You might post an example for the ".cells"-files you work on because i sense that for what you do there should be an easier solution than the one you found. Most probably some lines of awk will do everything your script does but ten times faster.

My suggestion regarding shells is: NEVER EVER USE CSH - it can't be stressed enough. csh is unpredictable as hell and strongly discouraged for interactive use, for scripting it is outright verboten!

If you want to write shell scripts use a POSIX-compatible shell. There are two: Korn shell (ksh) and Bourne-Again-Shell (bash). IMHO Korn Shell is better suited for scripting, but this may be personal taste as much as professional opinion. Any of the two will be suited for what your needs seem to be. It should be mentioned that the two shells languages differ only in details and if you know one you know most of the other. For what i think you want to do they are almost identical.

I do not know any bash books. For Korn Shell i can wholeheartedly recommend Barry Rosenbergs book "Hands-On KornShell93 Programming".

I hope this helps.

bakunin

Thanks you so much bakunin, we had our eid celebrations yesterday, that's why I'm posting late. I have changed my shell into bash so I think life will be a little bit easier even for a newbie like me. Thanks again and eid mubarak :smiley: