(bash) Script Processes in Parallel

Shell Programming and Scripting
1

Hello all,

I tried to parralise my treatments but after a while 'ps -ef' display all child process <defunct> (zombie)

Parent bash script to process all files (>100000) in directory:

for filename in /Data/*.txt; do
   ./child_pprocess.sh $filename &
done

exit(0)

I understand that the parent process does not capture the end of each child process.

But I do not know how to do it.
does anyone have a suggestion ?

Thank you in advance

2

You cannot create 100000 simultaneous child processes. Limit it to 20. And step through all of your files 20 at a time.

#/bin/bash   
cnt=0
for filename in /Data/*.txt; 
do
   ./child_pprocess.sh $filename &
    cnt=$(( $cnt + 1 ))
    if [ $(( $cnt % 20 )) -eq 0 ] ; then
       wait    # wait for all 20 to finish
    fi
done
wait   # you can get here with a few processes running, so wait for all of them
1 Like