parallel excution for 2000 files.

Shell Programming and Scripting
1 
Hi,

I have a function abc(). i am calling the function 9 times. it is working fine and completed the script execution 10 hours.


input files:
CUSTOMER_INFO_1111_12345.csv
CUSTOMER_INFO_1222_12345.csv
CUSTOMER_INFO_1322_12345.csv
CUSTOMER_INFO_1333_12345.csv
CUSTOMER_INFO_1151_12345.csv
CUSTOMER_INFO_2221_12345.csv
CUSTOMER_INFO_3333_12345.csv
CUSTOMER_INFO_4444_12345.csv

like that i have 2000 files are present inthe source directory.

abc()
{
#Calculations are added in the function
for file in `ls |egrep '(CUSTOMER_INFO_.*_'$Flag'.*csv$)'`
do
tail -1 $file1| awk '/^"9"/'  > new_$file
done
}

Flag=1
abc $p1 $p2 $p3 $p4  $Flag &
Flag=2
abc $p1 $p2 $p3 $p4  $Flag &
Flag=3
abc $p1 $p2 $p3 $p4  $Flag &
Flag=4
abc $p1 $p2 $p3 $p4  $Flag &
Flag=5
abc $p1 $p2 $p3 $p4  $Flag &
Flag=6
abc $p1 $p2 $p3 $p4  $Flag &
Flag=7
abc $p1 $p2 $p3 $p4  $Flag &
Flag=8
abc $p1 $p2 $p3 $p4  $Flag &
Flag=9
abc $p1 $p2 $p3 $p4  $Flag &

wait

echo "script completed"

Currently execution based on flag and at a time 9 files are executed. i want to run the script for all the 2000 files at the same time. 
when i run the scipt all the 2000 files get start for the excution. all the files are executed in parallel.
i am unable to execute files at a time.
2

Since these are triggered from the script paralley, Check the ulimit for sub background processes in your system. If ulimit value is less than 2000, I dont think so its possibkle for you to execute.

3 
ulimit -a

time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        4194304
memory(kbytes)       unlimited
coredump(blocks)     10485755
nofiles(descriptors) unlimited


as i can see that is unlimted in file(blocks) that means i can run unlimited files.
4

Slightly off topic.
The routine generates $file not $file1 .

This construct is likely to give trouble with 2000 files because the command line will become very long. The syntax shown is not valid in my "egrep" so it is difficult to suggest an alternative without knowing what selection of files was intended.

On topic: The kernel parameter for the maximum number of concurrent processes by a single user will be relevant. This value is not displayed by the "ulimit" command and is Operating System specific.

5 
touch the files 

CUSTOMER_INFO_12345_1111.csv
CUSTOMER_INFO_12345_2222.csv
CUSTOMER_INFO_12345_2234.csv
CUSTOMER_INFO_12345_3456.csv
CUSTOMER_INFO_12345_5553.csv
CUSTOMER_INFO_12345_6661.csv
CUSTOMER_INFO_12345_7752.csv

then the below command.

ls |egrep '(CUSTOMER_INFO_.*_1.*csv$)'
CUSTOMER_INFO_12345_1111.csv
ls |egrep '(CUSTOMER_INFO_.*_3.*csv$)'
CUSTOMER_INFO_12345_3456.csv

i am able to fetch the files. When you see the below command i am fetching two files.

ls |egrep '(CUSTOMER_INFO_.*_2.*csv$)'

CUSTOMER_INFO_12345_2222.csv
CUSTOMER_INFO_12345_2234.csv


the function should run parallely for the both files. Currently it is running for CUSTOMER_INFO_12345_2222.csv then followed by CUSTOMER_INFO_12345_2234.csv.

for the above command extracting for file names which is starts with 2.

Any help grealy appriciated.