I ran this script in AIX 5L environment and getting an error
usr/bin/ls: 0403-027 The parameter list is too long
Our administrator had increased the maxium allowable size of the ARG/ENV list but it still doesn't work.
I have tested the command in red below in the unix prompt and it works just fine, but why it failed from the shell programming script. Please advice.
scpdir=/$ROOTDIR/scp/inbox/
cd $scpdir
pntcnt=`find . -name 'PNT.*' -type f | wc -l`
if [[ $pntcnt -gt 0 ]] then
for gfile in `ls -1 /$ROOTDIR/scp/inbox/PNT.2*`
do
gline=`sed '1q' $gfile`
x=`echo "$gline" | awk '{ print substr( $0, 59, 1 ) }'`
filename=`echo "$gfile" | awk '{ print substr( $0, 20, 28 ) }'`
if [ "$x" -eq "0" -o "$x" -eq "1" ]; then
mv /$ROOTDIR/scp/inbox/$filename /$ROOTDIR/scp/inbox/string1/$filename
elif [ "$x" -eq "2" -o "$x" -eq "3" ]; then
mv /$ROOTDIR/scp/inbox/$filename /$ROOTDIR/scp/inbox/string2/$filename
elif [ "$x" -eq "4" -o "$x" -eq "5" ]; then
mv /$ROOTDIR/scp/inbox/$filename /$ROOTDIR/scp/inbox/string3/$filename
elif [ "$x" -eq "6" -o "$x" -eq "7" ]; then
mv /$ROOTDIR/scp/inbox/$filename /$ROOTDIR/scp/inbox/string4/$filename
elif [ "$x" -eq "8" -o "$x" -eq "9" ]; then
mv /$ROOTDIR/scp/inbox/$filename /$ROOTDIR/scp/inbox/string5/$filename
fi
done
fi
gfile is just a variable... I just used the same variables that you pasted in the actual piece of code...
for gfile in `ls -1 /$ROOTDIR/scp/inbox/PNT.2*`
Basically filesToProcess.tmp would be a temp file with the names of all the files that you want to rename/move... When its processed in the while loop, each file name is read into the variable gfile and then that variable is used down the line in the code...
Thank you for your clarification. Because I can not use the line code
ls -1 /$ROOTDIR/scp/inbox/PNT.2* > filesToProcess.tmp
due to an error "0403-027 The parameter list is too long". Then I have to replace it with
find . -name "PNT.2*" -print | ls -1 /$ROOTDIR/scp/inbox/ > FilesToProcess.tmp.
In the end result, it moved all the PNT.* files from the current directory plus all the PNT files from sub-directories which it should not. Is there a way that I just move those PNT files from the current directory but not the sub-directories?
Thanks
scpdir=/$ROOTDIR/scp/inbox/
cd $scpdir
pntcnt=`find . -name "PNT.*" -print | wc -l`
if [[ $pntcnt -gt 0 ]] then
find . -name "PNT.2*" -print | ls -1 /$ROOTDIR/scp/inbox/ > FilesToProcess.tmp
while read gfile
do
gline=`sed '1q' $gfile`
x=`echo "$gline" | awk '{ print substr( $0, 59, 1 ) }'`
filename=`echo "$gfile" | awk '{ print substr( $0, 1, 28 ) }'`
if [ "$x" -eq "0" -o "$x" -eq "1" ]; then
mv /$ROOTDIR/scp/inbox/$filename /$ROOTDIR/scp/inbox/string1/$filename
elif [ "$x" -eq "2" -o "$x" -eq "3" ]; then
mv /$ROOTDIR/scp/inbox/$filename /$ROOTDIR/scp/inbox/string2/$filename
elif [ "$x" -eq "4" -o "$x" -eq "5" ]; then
mv /$ROOTDIR/scp/inbox/$filename /$ROOTDIR/scp/inbox/string3/$filename
elif [ "$x" -eq "6" -o "$x" -eq "7" ]; then
mv /$ROOTDIR/scp/inbox/$filename /$ROOTDIR/scp/inbox/string4/$filename
elif [ "$x" -eq "8" -o "$x" -eq "9" ]; then
mv /$ROOTDIR/scp/inbox/$filename /$ROOTDIR/scp/inbox/string5/$filename
fi
done<FilesToProcess.tmp
rm -f FilesToProcess.tmp
fi
To avoid the problem of the "for" line getting to long use "while" not "for". We can then deal with any number of files.
If we redirect the error channel from "ls" we stop "ls" failing if there are no matching files. We now don't need to count the files or deal with the problem that the "find" actualy counts the files in the subdirectories too.
We can replace:
pntcnt=`find . -name 'PNT.*' -type f | wc -l`
if [[ $pntcnt -gt 0 ]] then
for gfile in `ls -1 /$ROOTDIR/scp/inbox/PNT.2*`
do
With:
ls -1d /$ROOTDIR/scp/inbox/PNT.2* 2>/dev/null | while read gfile
do
No need to use ls in for, because shell file generation can do it. Filegenaration is handled by shell, not by command (*, ?, ...). Maybe number of arguments is not problem for shell. If it's, then you need tmp-file.
This version try to use shell built-in commands = much faster as use external commands like awk, sed, ...
If I understood your needs, here is version N. Works with ksh, bash, posix-sh, ...
cd /$ROOTDIR/scp/inbox/
# no need to count number of files, it can be 0, no problem, but we must check it
for gfile in PNT.2*
do
[ "$gfile" = "PNT.2*" ] && break # no files, break the for loop
# read 1st line from file
read gline < $gfile
# substr without awk help, using shell built-in properties
x=${gline:58:1} # 1st char index is 0
# and if you like to run faster then no need in this case to use elif
# if it's true, do it and continue = skip to the next value
filename=${gfile:0:28}
if [ "$x" -eq "0" -o "$x" -eq "1" ]; then
mv /$ROOTDIR/scp/inbox/$filename /$ROOTDIR/scp/inbox/string1/$filename
continue
fi
if [ "$x" -eq "2" -o "$x" -eq "3" ]; then
mv /$ROOTDIR/scp/inbox/$filename /$ROOTDIR/scp/inbox/string2/$filename
continue
fi
# you can write also
[ "$x" -eq "4" -o "$x" -eq "5" ] && mv /$ROOTDIR/scp/inbox/$filename /$ROOTDIR/scp/inbox/string3/$filename && continue
[ "$x" -eq "6" -o "$x" -eq "7" ] && mv /$ROOTDIR/scp/inbox/$filename /$ROOTDIR/scp/inbox/string4/$filename && continue
done
In this example maybe case is the most beautiful (my opinion).
...
case "$x" in
0|1) mv ... ;;
2|3) mv ... ;;
...
*) ;; # default = what is this ?
esac
...
I agree that the "case" construct is more readable.
There are many posts on this forum where the "for" construct breaks because the list causes the "for" statement to be too long ... or any filename contains spaces. Depends on local conditions.
I think that problem is not for, problem is ls.
I just tested with 62000 files in one directory. ls *.txt not work but
for f in *.txt
works fine.
If you use ksh (software download selections), then filegeneration limit is maybe some huge value, but 62000 files was okay.
for also works fine with filenames which have spaces and other special characters.
Why so many use
for f in $(ls *.txt) ?
for f in *.txt is enough, no need to use with ls
Maybe dos history ?
In sh-shells *, ?, ... are handled by shell, not by command, so you can use those marks with any commands like
print *.txt
You are right. Even with a modern shell it is quite easy to exceed the maximum size of a variable or the maximum length of a command line. The dodgy syntax includes:
for f in $(ls *.txt)
for f in `ls *txt`
f=$(ls *.txt)
f=`ls *.txt`
I too wonder why so many are expanding "ls" on the command line.