Using freq in script

Hi,

I have a requirement where I need to check a column data in a fixed length file.

Sample file.

1 10 20150201 TEST1 TEST1
2 20 20140201 TEST2 TEST2
3 30 20150810 TEST3 TEST3
4 10 20150201 TEST4 TEST4
5 10 20150201 TEST5 TEST5
6 30        0 TEST6 TEST6
7 20 20140201 TEST7 TEST7
8 10 20140201 TEST8 TEST8
9 20 20120609 TEST9 TEST9
1  0 20120609 TEST0 TEST0
2 20 20150201 TEST2 TEST2
3  0        0 TEST3 TEST3
4  0        0 TEST4 TEST4
5  0 20150201 TEST5 TEST5

The characters 6-13 is a date column for which the value has to be 8 digits. I don't need to validate the date format that its an actual date, only that the value is 8 digits.

I am able to do a freq on the file which will tell me the counts per value:

/>freq inputfile.txt 6 13
inputfile.txt
       0  3
20120609  2
20140201  3
20150201  5
20150810  1
Total    14

Can anyone help me as to how this can be checked in a shell script?

Thanks.

Not sure what output you are after. Here I report if file contains any invalid rows:

if grep -qvE '^.{5}[0-9]{6}' inputfile.txt
then
    echo "File is invalid"
else
    echo "File is valid or empty"
fi

If you are reading the file in line by line in a loop, you could add this type of thing too:-

i=0
while read col1 col2 col3 col4 col5
do
   ((i=$i+1))
   if [ "${#col3}" -ne 8 ]
   then
      echo "Error on line $i"
      sed -n${i}p file           # Read out the illegal line directly from the file
   fi
done < file

Does that give you an option to the above suggestion? Which way suits your existing code?

Robin

In Chubler_XL's proposal, 8 digits should be checked, and you could stop after the first invalid entry, saving some time on large files:

if grep -qvEm 1 '^.{5}[0-9]{8}' inputfile.txt

Thanks Chubler_XL, Robin and RudiC. This is really helpful.

I changed Chubler_XL's code to use 8 characters and it is working perfectly.

Robin - Thanks for your suggestion. The test file I have provided here is just a sample. The original file have over 150 columns and is a big file with around 1MM records per cycle.
I think the if statement will be better for this file so we dont' have to read all records.

RudiC - your suggestion on exit when you hit an invalid entry is really good. That way we don't have to go through the entire file.

Nice pickup on the number of digits RadiC, must have miscounted there.

I'm fairly sure that -q will exit on first matching record so -m 1 will not add any more efficiency.

Yes - should have read the entire -q entry in man grep ...