KSH SHELL: problem calculation number of lines inside compressed file

Shell Programming and Scripting
1

Hi Gurus,
I am working with a korn shell script to simplify some operations of calculation number of lines inside compressed file.
The called function (inside a cycle) is the following:

 
#########################################
# F.ne: CheckCount
#########################################
function CheckCount
{
 
CDRHDTR=2
CDRI=0
for xx in ${DIRLIST}
do
 
        cd ${OUTDIR_MOD1}${xx}
        LISTO=`cat ${dir_log}LIST_1_$RepDate.tmp`
        for fileo in $LISTO
        do
        if [[ ! -f "${fileo}" ]]
        then
        CDRO_TOT=0
        file_upd=`echo $fileo | cut -d"*" -f1`
        UpdateOut $CDRO_TOT ${file_upd}"%"
        else
        CDRO=`gzcat ${fileo} | wc -l | awk '{print $1}'`
        CDRO_TOT=`echo "$CDRO - $CDRHDTR" | bc`
        file_upd=`echo $fileo | cut -d"*" -f1`
        UpdateOut $CDRO_TOT ${file_upd}
        fi
        done
done

Unfortunately the performances are very bad, and in some cases straight
it performs a wrong calculation!!!

an idea to optimize the job and to make sure in the calculation?

Thanks,
Germano

2

Later in the script there is code that suggests that $fileo can contain asterisk characters. Very important that we look at what gets put into this variable.

What is in ${DIRLIST} ?
What is in ${OUTDIR_MOD1} ?
What is in ${dir_log} ?
What is in the file LIST_1_$RepDate.tmp ?

How many directories, hoe many files, how big?

3

I recover the parameters from a file of configuration in this way:

##### Constants
# recover properties from file
. ../cfg/init.cfg

these are the values:

 
DIRLIST="20101227 20101228 20101229"
OUTDIR_MOD1="/data/remote/filter/output/"
dir_log="="/data/remote/filter/log/"

works on 44 directory and round 660000 file(compressed) what they contain 1500 rows each.
LIST_1_$RepDate.tmp (LIST_1_20110126_102312.tmp) it contains the list of the files
to elaborate, this is a part of the content:

 
NC_AN01MSC_GSM_20101227_5493*
NC_AN01MSC_GSM_20101227_5507*
NC_AN01MSC_GSM_20101227_5508*

the character "*" to end line he has on purpose put, because it serves in this format
for an following operation sql:

#########################################
# F.ne: Updateout
#########################################
UpdateOut()
{
isql -U$USER_DB -P$PWD_OPER -S$SERVER -D$DB_OPER w1000 << THEEND >> ${LogFILENAME}
set nocount on
go
UPDATE $DB_OPER..KPI_FILTERED set NUM_CDR_OUTPUT=${1}
WHERE FILE_NAME LIKE '${2}'
go
quit
THEEND
}

all this correctly works...the variables arrive cleaning up, the problem I think both
this operation:

CDRO_TOT=`echo "$CDRO - $CDRHDTR" | bc`

do you know a faster and effective method(no expr)?

I in advance thank you,
Germano

4

What Operating System and version do you have? Many would not process this script with 660,000 filenames on the command line. Or are you testing with a small number of directories and a small number of files.
Unclear whether each of the 44 directories contains 660,000 files with their names listed in the files list file ... or whether there are just 660,000 files spread across 44 directories.

There appear to be much bigger problems with this script than the speed of arithmetic in "bc".

We must understand whether the names of the actual files on disc have this trailing asterisk character. The way the script is written makes me think that they do.
What does the directory list for say these three files look like?
NC_AN01MSC_GSM_20101227_5493*
NC_AN01MSC_GSM_20101227_5507*
NC_AN01MSC_GSM_20101227_5508*

Also is the number at the end of the file name variable length?
For example are all these names valid?
NC_AN01MSC_GSM_20101227_5*
NC_AN01MSC_GSM_20101227_54*
NC_AN01MSC_GSM_20101227_549*
NC_AN01MSC_GSM_20101227_5493*

The main design issue is that "isql" is loaded for every file listed in the files list file. This will be very slow.

Once we get it clear how many files there are and what their names are, we can look at whether this job is feasible in Shell.