Loop counter resets by itself

Hi!

Can anyone explain this? The counter CDR_count should go on forever, but it suddenly resets at every step of the FOR loop (I know this because $_file has exactly 378 records). However, the counter reset is OUTSIDE the FOR loop. What's going on??

#!/bin/bash

if [ "$1" = "" ] || [ "$2" = "" ]; then
   echo
   echo "USAGE: ./CCN_CDR_parser.sh <CCN CDR files - path & wildcard allowed> <MSISDN>"
   echo
   exit
fi

echo
echo "Searching CDRs for MSISDN $2 in:"
echo "$1"
echo

file_count=0
file_total=`ls -1 $1 | wc -l | tr -d " "`
CDR_count=0
CDR_match=0
echo > CDR.tmp

for _file in `ls -1 $1`
do
  echo -ne "Files processed: $file_count / $file_total       CDRs match : $CDR_match / $CDR_count"\\r
  file_count=`expr $file_count + 1`

     cat $_file | while read _line
     do

             if [ "$_line" = "CDRCCN.ChargingDataOutputRecord.onlineCreditControlRecord" ]; then

                CDR_count=`expr $CDR_count + 1`
                echo "Files processed: $file_count / $file_total       CDRs match : $CDR_match / $CDR_count"

                if [ "`grep "servedAccount : '$2'" CDR.tmp`" != "" ]; then
                   CDR_match=`expr $CDR_match + 1`
                   mv CDR.tmp CDR$CDR_match.txt
                   echo -ne "Files processed: $file_count / $file_total       CDRs match : $CDR_match / $CDR_count"\\r
                fi

                CDR_start="true"
             else
                CDR_start="false"
             fi


             if [ "$CDR_start" = "true" ]; then
                echo $_line > CDR.tmp
             else
                echo $_line >> CDR.tmp
             fi

      done
done

echo
echo

Files processed: 23 / 1192       CDRs match : 0 / 373
Files processed: 23 / 1192       CDRs match : 0 / 374
Files processed: 23 / 1192       CDRs match : 0 / 375
Files processed: 23 / 1192       CDRs match : 0 / 376
Files processed: 23 / 1192       CDRs match : 0 / 377
Files processed: 23 / 1192       CDRs match : 0 / 378
Files processed: 24 / 1192       CDRs match : 0 / 1
Files processed: 24 / 1192       CDRs match : 0 / 2
Files processed: 24 / 1192       CDRs match : 0 / 3
Files processed: 24 / 1192       CDRs match : 0 / 4

CDR_count=`expr $CDR_count + 1`

Is depricated, try instead:

CDR_count=$(( $CDR_count + 1 ))

Likewise for CDR_match.

Other than that (which could, or not, be the cause) the counting should work.

hth

EDIT:
You could add something like

echo running here

Inside that block, to see if its actualy executed.

Or just start the script like:

$SHELL -x ./scriptname

It's gotta be the craziest bug of my carrier.

     cat $_file | while read _line
     do

             if [ "$_line" = "CDRCCN.ChargingDataOutputRecord.onlineCreditControlRecord" ]; then
#                CDR_count=`expr $CDR_count + 1`
                CDR_count=$(( $CDR_count + 1 ))

                echo "Files processed: $file_count / $file_total       CDRs match : $CDR_match / $CDR_count"

                if [ "`grep "servedAccount : '$2'" CDR.tmp`" != "" ]; then
#                   CDR_match=`expr $CDR_match + 1`
                   CDR_match=$(( $CDR_match + 1 ))
                   mv CDR.tmp CDR$CDR_match.txt
                   echo -ne "Files processed: $file_count / $file_total       CDRs match : $CDR_match / $CDR_count"\\r
                fi

                CDR_start="true"
             else
                CDR_start="false"
             fi

             if [ "$CDR_start" = "true" ]; then
                echo $_line > CDR.tmp
             else
                echo $_line >> CDR.tmp
             fi
echo test1 CDR_count=$CDR_count
      done
echo test2 CDR_count=$CDR_count

test1 CDR_count=379
test1 CDR_count=379
test1 CDR_count=379
test1 CDR_count=379
test2 CDR_count=0
Files processed: 2 / 2       CDRs match : 0 / 1
test1 CDR_count=1
test1 CDR_count=1
test1 CDR_count=1
test1 CDR_count=1

change:

cat $_file | while read _line
do

done

to

while read _line
do

done < $_file

Oh man, this works thanks a lot. Any reasonable explanation for this??

Using a pipe puts the loop into a subshell. Any variable changes inside it aren't reflected outside the loop, and once the loop finishes, the subshell ceases to exist. It's kind of like the difference between running a script and sourcing it.

It's also a good reason to get out of the habit of useless use of cat.

Wow, makes sense now. Thanks a lot

First, you don't need -1 when the output of ls is not going to a terminal.

Second, the command will fail if $1 contains whitespace or other pathological characters.

Third, there is no need for tr; it has been a long time since wc's output was preceded by a space.
And if it were, you could remove it with:

file_total=$(( $(ls "$1" | wc -l) ))

Or:

file_total=$(ls "$1" | wc -l)
file_total=${file_total// /}

But you can do it without any external commands:

file_list=( "$1"/* )
file_total=${#file_list[@]}

See the first 2 comments above.

It can fail even if you quote $1.

(( ++file_count ))

UUOC and unquoted variable

(( ++CDR_count ))

if grep "servedAccount : '$2'" CDR.tmp; then

(( ++CDR_match ))

Also, if you want to retain the value of variables set in the last process in a pipeline, use:

shopt -s lastpipe

Always a pleasure to read your posts! Nice way of removing spaces by using arithmetic expansion.

Other benefits of this method are that it also correctly counts file names that contain newlines (whereas ls|wc -l does not) and that there are no command line length limitations..