Test File Reading & Validation using Shell script

Shell Programming and Scripting
1

Please help develop script for below requirement

-------Sample file-------------------------------
HSVSHOSTRECON 20090115011817BP            
DARMAR60064966247003504720000000000000000000066626000000000000133000003D003463001332    
DARMAR60064966247008334930000000000000000000066626000000000000191000003D006888001911    
DARMAR60064966247023430040000000000000000000066626000000000000237000003D010451002371    
DARMAR60064966247018096900000000000000000000066626000000000000358000003D000069003581    
DARMAR60064966247026650830000000000000000000066626000000000000358000003D011186003581    
DARMAR60064966247028148300000000000000000000066626000000000000358000003D009154003581    
DARMAR60064966247019083510000000000000000000066626000000000000374000004D030784003741    
DARMAR60064966247018409190000000000000000000066626000000000000428000003D010969004281    
DARMAR60064966247021076490000000000000000000066626000000000000442000003D005373004421    
DARMAR60064966247028097310000000000000000000066626000000000000442000003D008266004421
DARMAR60064966247028097310000000000000000000066626000000000000442000003D008266004421
T00000011                                        
-----------------------------------------------

Here is the List of Validation. We need to send mail for each validation failure. First Char is Type of Record Say Header/Detail/Trailer

  • 1. The first record in the file must be the header record.
  • 2. The file can have at most one header record.
  • 3. The date on the header should match the date received (Note: If a file in error is retransmitted the next day, this may not be so. The purpose of this edit is to make sure that we are not picking up a duplicate of an earlier file.)
  • 4. The last record in the file must be a trailer record. (This means that the file can have at most one trailer record, and that there should be no header or detail records following the trailer record.)
  • 5. Every record between the header record and the trailer record must be a detail record.
  • 6. The trailer count should match the calculated number of detail records in the file.

Regards
Niraj Bhatt

2

I have the following queries...

  1. Is the file name same :slight_smile: i ment for the one validation has to be done?

  2. The date feild in the first line can be validated with system date :b:?

  3. The first line i ment header is always with time stamp and footer is always with no. of records?

  4. Will ther be a common patern matcher for the record... in teh below case like "DARMAR" :slight_smile:

Regards
Bips...

3

Hi Bips,

Please see below answers

  1. Is the file name same :slight_smile: i ment for the one validation has to be done?
    it will be <filename><receivedate>.txt format.

  2. The date feild in the first line can be validated with system date :b:?
    Yes

  3. The first line i ment header is always with time stamp and footer is always with no. of records?
    Yes...Header will be date & time stamp and footer will be with total records

  4. Will ther be a common patern matcher for the record... in teh below case like "DARMAR" :slight_smile:

D stands for detail...after that records will be changed.

4

Some more inputs.

First Char is Type of Record Say Header/Detail/Trailer
Last records contains number of detail records i.e. 11 it can be 105 or 1230 also but position will be same always.
First record has date which is 20090115011817 in yyyymmddhhmmss format position in header will be same always

5

Hey Niraj,

Kindly execute the below script in the following pattern:

cmd$>sh script.name <file.nameto be validated>

below is script.name
---------

#!/bin/sh
X=`date +%Y%m%d`
#W=`cat X`
head -1 $1 > Y
Z=`cat Y | grep -i $X | wc -l`
if [ "$Z" -eq 1 ]
then
         echo "Record generated today found"
          temp1=`tail -1 $1 | wc -c`
          length=`expr $temp1 - 1`
          t=0
          Target_Record=0
          s=1
          while [ $length -gt $s ]
          do
                  len=`expr $length - $t`
                  x=`tail -1 $1| cut -c $len`
                  if [ $t -le 0 ]
                  then
                         y=0
                         Target_Record=`expr $y + $Target_Record + $x`
                  else
                         y=`expr $x \* 10`
                         Target_Record=`expr $y + $Target_Record `
                  fi
                  s=`expr $s + 1`
                  t=`expr $t + 1`
           done
           echo "Expected enties as per footer:"$Target_Record
           temp_1=`cat $1|wc -l`
           temp_5=`expr $temp_1 - 2`
           if [ $temp_5 -eq $Target_Record ]
           then
               echo "record is perfect"
           else
               echo "record is corrept"
           fi
else
           echo "record not of todays date"
fi
exit 1

---------------

i have assumed that the total count of individual records is equal to total lines present in the record minus first and teh last line....

i guess this will help you....

Regards,
Bips :b:.

6

Hi Bips.

Can I request you to please add some comments so that It is more clear to understand ?

Did you cover all 6 Validation ?

Regards
Niraj Bhatt

7

Hi Niraj,

The script does the following:

  1. Checks if the files header corresponds to the current date.
    else the output is "record not of todays date"
  2. Checks if the file has more than one header it wil fail with error "record is corrept"
  3. Checks the date in the header with the system date if ther is a missmatch it wil fail with "record not of todays date"
  4. If there are more than one trailer record it wil give output "record is corrept"
  5. Every record between the header and trailer is considered as a Detail record. I was not able to check the consistancy of that as u did not give me any standard on that.
  6. The trailer count will match the records found if sucessful it wil give message "record is perfect" else fails with the message its corrept.

I have not considered output to be redirected to a file.
Example:

Bips> cat test
HSVSHOSTRECON 20090219011817BP
DARMAR60064966247003504720000000000000000000066626000000000000133000003D003463001332
DARMAR60064966247008334930000000000000000000066626000000000000191000003D006888001911
DARMAR60064966247023430040000000000000000000066626000000000000237000003D010451002371
DARMAR60064966247018096900000000000000000000066626000000000000358000003D000069003581
DARMAR60064966247026650830000000000000000000066626000000000000358000003D011186003581
T00000005
Bips> sh sample test
Record generated today found
Expected enties as per footer:5

record is perfect
----
2nd case :
---

Bips> cat test
HSVSHOSTRECON 20090219011817BP
DARMAR60064966247003504720000000000000000000066626000000000000133000003D003463001332
DARMAR60064966247008334930000000000000000000066626000000000000191000003D006888001911
DARMAR60064966247023430040000000000000000000066626000000000000237000003D010451002371
DARMAR60064966247018096900000000000000000000066626000000000000358000003D000069003581
DARMAR60064966247026650830000000000000000000066626000000000000358000003D011186003581
T00000005
T00000005
Bips> sh sample test
Record generated today found
Expected enties as per footer:5

record is corrupt
---
I guess its clear now...

Regards,
Bips.:b:

8

Hi Dip,

"Every record between the header and trailer is considered as a Detail record. I was not able to check the consistancy of that as u did not give me any standard on that."

Can you please let me know what you need to know about consistency ?

All the records between header and Trailer must be detail record. If there are more than one header or trailer record than file is corrupt.

Regards
Niraj

9

Try this :

$ validate.sc your_file_name

validate.sc is the following:

#!/bin/ksh
today=`date +%Y%m%d`
TRAILER=false
HEADER=false
DETAIL=0
while read LINE
do
#
#  Validating 1st record is Header + header's date
#
   FIRST_CAR=`echo $LINE | cut -c1`
   if [ $HEADER = "false" -a $FIRST_CAR = "H" ]
   then
      HEADER=true
      echo $LINE | grep $today > /dev/null
      if [ $? -eq 1 ]
         echo "Header record not from today"
         exit 1
      fi
   else
#
#  Validating detail record or trailer recors
#
      if [ $HEADER = "true" ]
      then
         if [ $FIRST_CAR = "D" ]
         then
            if [ $TRAILER = "false" ]
            then
               DETAIL=`expr $DETAIL + 1`
            else
               echo "File is corrupt - Detail record found after a trailer record"
               exit 1
            fi
         else
            if [ $FIRST_CAR = "T" -a $TRAILER = "false" ]
            then
               TRAILER=true
               TOT=`echo $LINE | cut -c2-`
               if [ $DETAIL -ne $TOT ]
               then
                  echo "File is corrupt - Number of detail record mismatch"
                  exit 1
               fi
            else
               echo "File is corrupt - Invalid record or more than one header/trailer record"
               exit 1
            fi
         fi
      else
          echo "File is corrupt - No header record found"
          exit 1
      fi
   fi
done < $1
if [ "$TRAILER" = "false" ]
then
   echo "File is corrupt - No trailer record found"
   exit 1
else
   echo "File seems to be Ok"
fi
10

First UEC (Unnecessary External Command). External commands slow a script considerably, especially when in a loop.

FIRST_CAR=${LINE%"${LINE#?}"}

Second UEC.

case $LINE in
     *"$today"*) ;;
     *) echo "Header record not from today" 
         exit 1 ;;
esac

Third UEC.

DETAIL=$(( $DETAIL + 1 ))

Fourth UEC.

11

I presume that would be the fourth UEC

What should it be ?

thx

12 
 TOT=${LINE#?}

Regards

13

Hi Dip,

I am also getting error as below at TOT variable

./validate.sh[9]: syntax error at line 47 : ``' unmatched

Regards

14

Hi ce9888

I am getting below eeror with example I gave.

File is corrupt - Invalid record or more than one Header/Trailer record"

There is only one Header and trailer still I am getting that error.

Can you please advise ?

Regards

15

Hi ce9888 & Dip,

Please ignore previous two replies...I got them worked.