Find command when there exist many files

Shell Programming and Scripting
1

I want to run find and wondering if it struggles when there are many files. I have tried
and does not seem to complain.

Is this correct?

2

It won't 'struggle' when there are many files.. it just might take longer to finish (depending on where it's looking). There's no chance of an 'arg list too long'-type error, as with, for example ls *, or similar commands, where the shell tries to expand *. find's often the best command to use in many situations where there are very many files involved.

3

I'm pretty sure it would complain when running into difficulties. What "struggles" do you expect?

4

For example I cannot put all the files in a string when running a script as the amount of files is enormous. Also I cannot use a file display program such as Nautilus, the screen takes too long to load all the files.

I then need to run a program with each file as input to a program called mseed2sac and
may create multiple files for each file passed.

Now I want the script to give me a percentage so that user will know the progress. Ideally
I do not want that every calculation is printed on a new line. The following does not seen to
work well.

# Counts the number of files to process
flcnt=$(find . -type f | wc -l)
echo "flcnt: $flcnt" 

i=0
for f in $(find . -type f); do
  #echo "+ $f"
  i=$((i++))
  percent=$(awk "BEGIN { pc=100*${i}/${flcnt}; j=int(pc); print (pc-j<0.5)?j:j+1 }")
  echo "$percent"
  #${dir_mseed2sac}/mseed2sac $f
done
5

The command:

  i=$((i++))

does the following things, in sequence:

  1. $((i++)) part 1: save the initial value of i ,
  2. $((i++)) part 2: increment the value of i ,
  3. $((i++)) part 1: return the initial value of i , and
  4. i=... : assign the returned value to i .

These four steps together assign the initial value of i to i discarding the increment that was performed in step 2.

To get what I think you're trying to do, try changing that line to one of the following:

  i=$((i+1))
  i=$((++i))
  ((i++))
  ((++i))
  ((i=i+1))

The first two of these will work with any POSIX-conforming shell. The last three will work in ksh , some versions of bash , and might also work in some other shells.

Furthermore, anytime you use find to get a list of files, you have to assume that any filename allowed by the underlying filesystem might pop us in find 's output. You should write your code to assume that a pathname returned by find might contain whitespace or other problematic characters. So, a command like:

  #${dir_mseed2sac}/mseed2sac $f

(assuming that you intend to remove the # making it a comment at some later time, should instead be written as one of the following:

  #"${dir_mseed2sac}"/mseed2sac "$f"
  #"${dir_mseed2sac}/mseed2sac" "$f"
1 Like
6

I want to display the percentage when it changes by 3% and only print
then keeping the cursor on the same line.

Some suggestions would help me.

---------- Post updated at 07:03 AM ---------- Previous update was at 06:56 AM ----------

I have done it this way, which looks good

i=0
for f in $(find . -type f); do
  #echo "+ $f"
  i=$((i+1))
  percent=$(awk "BEGIN { pc=100*${i}/${flcnt}; j=int(pc); print (pc-j<0.5)?j:j+1 }")
  #echo "$percent"
  echo -e "\r${percent}%\c"
  #${dir_mseed2sac}/mseed2sac $f
done

---------- Post updated at 07:13 AM ---------- Previous update was at 07:03 AM ----------

The final thing I would like to do is have a progress bar enclosed by [ ]
and using stars to show the completion. The total number of stars should
be 60 for completion

Example:

[**********                                    ] 10% 
 

i=0
nstars=0
for f in $(find . -type f); do
  #echo "+ $f"
  i=$((i+1))
  percent=$(awk "BEGIN { pc=100*${i}/${flcnt}; j=int(pc); print (pc-j<0.5)?j:j+1 }")
  #echo "$percent"
  echo -e "\r${percent}%\c"
  nstars=$((percent*0.6))
  echo -e "[*****                         ]"
  #${dir_mseed2sac}/mseed2sac $f
done
7

How about - given you're using a recent bourne type shell (which you fail to mention) -

TOTFC=$(find . -type f | tee /tmp/WRK | wc -l)
while read FN
  do    printf "\r%4d %%" $((100 * ++i / TOTFC)) 
        #${dir_mseed2sac}/mseed2sac $FN
  done < /tmp/WRK
printf "\n"

EDIT: for the star bar, try

        printf -v XXX "%0*d" $((60 * ++i / TOTFC))
        printf "\r[%-60s]" "${XXX//0/*}"
8

I have found a progress bar on github but when I try the following it does not
do anything

source progress_bar.sh
progress_bar 300

GitHub - nachoparker/progress_bar.sh: Progress bar for the shell

---------- Post updated at 10:36 AM ---------- Previous update was at 08:55 AM ----------

I have now modified my script this way. It works properly, however I have a white
on black colour scheme on my terminal and I am noticing a white box scrolling along
the line all the time. Can this be fixed?

# Counts the number of files to process
totfcn=$(find . -type f | tee /tmp/wrk | wc -l)

i=0; j=0
while read fn; do    
  printf -v XXX "%0*d" $((60 * ++i / totfcn))
  printf "\r[%-60s]" "${XXX//0/*}"
  printf "%4d%%" $((100 * ++j / totfcn))
  #${dir_mseed2sac}/mseed2sac $FN
done < /tmp/wrk
printf "\n"
9

The white box is probably you input cursor you can hide/view it like this:

# Counts the number of files to process
totfcn=$(find . -type f | tee /tmp/wrk | wc -l)

# Hide cursor
tput civis

# Reveal cursor on program completion
trap 'tput cvvis' EXIT

i=0; j=0
while read fn; do    
  printf -v XXX "%0*d" $((60 * ++i / totfcn))
  printf "\r[%-60s]" "${XXX//0/*}"
  printf "%4d%%" $((100 * ++j / totfcn))
  #${dir_mseed2sac}/mseed2sac $FN
done < /tmp/wrk
printf "\n"
1 Like