File format check

UNIX for Beginners Q & A UNIX for Dummies Questions & Answers
1

How to check if file is in a given format?

For instance: if file records are delimeted with "|" ( pipes) and have exactly 26 fields?

File is pretty big (~3 mil reccords), so not sure if I have to check all records or just head/tail records or smth.

Any ideas are much much more than welcome!

2

If there is a possibility that some records are bad, you need to read the whole thing. If you are just identifying the format, you can read a few lines.
whole file scan

awk -F'|'  'BEGIN{ max=0} 
             { if (NF > max ) {max =NF} 
             END{ print "max fileds found=", max}' inputfile

partial:

head inputfile | awk -F'|'  'BEGIN{ max=0} 
             { if (NF > max ) {max =NF} 
             END{ print "max fileds found=", max}'
3

Thanks you,

That is definately the right way to do it!!!

4

What is wrong?

5

The single quotes are protecting $tmp from being variable expanded by the shell. Unquote around $tmp:

awk -F '|' '(NF!='$tmp'){print "Invalid line:", $0}' ../data/CUSTOMER.txt
6

Thanks MRC!

Another quick one:
What is the right awk syntax and how to write/redirect records into two files(good and bad):
if (NF!='$temp')
then write to the bad_file.txt
else write to the good_file.txt
fi

7

awk -F '|' '{if (NF=='$tmp') { print "Good line:", $0 >> "good_file.txt" } else { print "Invalid line:", $0 >> "bad_file.txt" }}'

8

Thanks again!!!

9

Or even without double '>':

awk -F\| '{print>((NF!=nf?"bad":"good")"_file.txt")}' nf="$tmp" infile
10

Right, >> is append, and > is replace.

The ternary operator is your friend in these simple cases. I started to use the ternary operator but then thought the OP might have more print lines or other operations necessary on either side of the condition, so gave way to the traditional if-then-else.

11

Yep,
I only wanted to point out that the AWK redirection is different from the shell redirection. From Effective AWK Programming:

Consider the following:

$ cat file
1 2 
1 2 3
1 2
1 2 3
1 2
$ awk '{print>((NF!=2?"more":"two")"_columns")}' file
$ head *_col*
==> more_columns <==
1 2 3
1 2 3

==> two_columns <==
1 2 
1 2
1 2
12

Understood. The redirection plumbing in awk is setup once, and subsequent > or >> redirections simply continue writing on the already open FD. In shell scripts, the FD is closed at the end of each command.