Validating a formatted message

Shell Programming and Scripting
1

I'm trying to write a shell script that will validate a format of a data file we recieve every night.

What I'm looking for is to ensure there are 8 pipes "|" before an email address and 9 pipes after the email address.

I was also looking to do this in one of two ways -
a) Grabbing random lines out of the files and then checking them. I don't have any idea on how to do this - I did a search on random number generators but my knowledge of scripting is slim.

b) Checking the whole file (which can be a max of 200k records).

Any help would be appreciated.

Thank you!!!

Brian

2

Post some sample lines.

3

Of the data file?

It's going to be something like this (at the bare minimum):

||||||||123456@abc.com|||||||||

There may be data in the columns, but only 1 email address (hence, only one @ sign).

Let me know if you need more information..

Brian

4

This should do it. The code looks odd, I know. But it will run pretty quickly.

#! /usr/bin/ksh
integer errors
errors=0
IFS=""
while read inputline ; do
        line="${inputline%\|*([!|])}"
        line="${line%\|*([!|])}"
        line="${line%\|*([!|])}"
        line="${line%\|*([!|])}"
        line="${line%\|*([!|])}"
        line="${line%\|*([!|])}"
        line="${line%\|*([!|])}"
        line="${line%\|*([!|])}"
        line="${line%%*([!|])}"
        line="${line#*([!|])\|}"
        line="${line#*([!|])\|}"
        line="${line#*([!|])\|}"
        line="${line#*([!|])\|}"
        line="${line#*([!|])\|}"
        line="${line#*([!|])\|}"
        line="${line#*([!|])\|}"
        line="${line##*([!|])}"
        if [[ "$line" != \|*([!|@])@*([!|@])\| ]] ; then
                echo error: "$inputline" 
                ((errors=errors+1))
        fi
done
((errors)) && echo $errors total errors
exit $(($errors!=0))
5

Thanks for your help - like I said my knowledge of scripting is very meager...

How would I go about setting it up to accept the input file to read through it? ie so I can call the shell script and feed it the name of the file to check as input?

Brian

6

Put those lines in file called checker or something. Get them exactly right. Close doesn't count. Then make it executable:
chmod u+x checker
Then run it:
./checker < inputfile

7

Thanks for your help, it runs like a champ!

Brian

8

If you absolutely must use a shell and nothing else, then Perderabo's solution is probably as good as you're going to get.

However, if you can use standard Unix programs and you have a nawk, this will execute faster (especially if you have 200K records every night).

#!/usr/bin/nawk -f
BEGIN { FS = "|" }
NF != 18 || !index($9, "@") { errors++ }
END { print errors, "errors"; exit errors != 0 }

You wanted 8 "|" before the email address and 9 after. Changing FS to "|" (a literal string) makes the field count be 18 on lines with 17 of these. But if there is a different number or if the 9th field (following the 8th "|") doesn't contain an "@", count the line as an error.

This doesn't check for all valid forms of email addresses --- that takes a parser. It also doesn't check for possible email addresses in other positions.

You could implement similar solutions in Ruby, Perl or (I presume) Python. You could actually massage the shell into doing something similar, but awk (or one of the others) will execute faster.

9

I'm sorry, I guess I should of made that clear - awk/nawk is fine..

Yours ran extremely quick against a 685k test file I had. While I don't get files > 200k more then once a night, I'd like to be safe and make sure it works.

We do have an email validator that someone else wrote before I got here - so I'm not worried about that. I was just worried about having the 8 pipes before and 8 pipes after the email address.

Is there an easy way in awk to have it copy the bad lines out to a seperate file?

I'm a junior admin here learning as I go along and would like to automate some of this stuff that my coworkers hadn't gotten a chance to do..

Brian

10

Try this:

#!/usr/bin/nawk -f
BEGIN { FS = "|" }
NF != 18 || !index($9, "@") { errors++ ; print }
END { print errors, "errors"; exit errors != 0 }

Call it this way (assuming you have put the above in a file called "findbadlines" and marked it executable).

findbadlines infile1 infile 2 [...] >badlinesfile

As with all code you find here (or anywhere else), you should understand it before you put your job on the line.

11

I figured out how to do it and got it working.

I'm making sure I understand exactly what it is the script is doing before I run with it. Makes me explaining it later on a little easier :slight_smile:

Thanks for all of your help

Brian