My boss has asked me to create a unix script to check header files vs data files and to send an email in case of any failure.
I have very little unix scripting experience and it was now long ago so I'm a bit concerned I wont be able to turn this around by end of day tomorrow.
I've taken his requirements and turned it into a kind of pseudocode to help me design the script but now the exact Unix commands to use for each piece will require a lot of googling and testing on my part.
I am hoping that some kind person with years of experience will be able to turn this into ksh script for me in a matter of minutes
--------------- Main Procedure
Go to temp folder
Check for a header file matching mask: F57DWHDR.yyyymmddhhmmss.CSV
If no header matching that mask <FAIL1>
Else <CHECKHEADER>
<MOVEFILES>
--------------- Sub Procedures
<FAILn>
Email failure
<CHECKHEADER>
Check for all records ending in '*' (wc -l vs grep | wc -l?)
If not equal <FAIL2>
Else <CHECKFILES>
<CHECKFILES>
(LOOP THROUGH ALL LINES)
Read line: Filename, count (cut -c?)
Check for file existance
If not exist <FAIL3>
Else <CHECKCOUNT>
(ENDLOOP)
<CHECKCOUNT>
Compare readcount to filereccount
if not equal <FAIL4>
<MOVEFILES>
Move file to current area already used by existing code
Here is an example header file, fixed width (50 C filename, 10 C count, ' *) = 62 chars.
Any help with this would be ever so much appreciated, but if you all have better things to do I wont be offended and will crack on with this tomorrow morning!
I think it would be easier if you phrased the requirements in plain English, supported by sample data. My imagination right now does not suffice to understand your pseudo code (e.g. what do you compare to what?).
Thanks for the quick reply, RudiC, sorry thought Pseudocode would help be unambiguous but here it is in English:
Checks existence of a header file matching mask F57DWHDR.yyyymmddhhmmss.CSV
Checks that header file to ensure every record ends in '*'
Loops through every line to check that
3a) the filename in the header file exists
3b) the record count in the file matched whats in the header file (eg file F4211OPN20130916041521.CSV has 41216 lines)
Any failure in the above results in an email being sent to say 'xyz@abc.com'
Thanks so much RudiC, will try to test this shortly but after a bit of googling I think I have understood what you are trying to do, please let me know if I have misunderstood any commands. I changed the continue to a break as in case of any failure in verification, processing should stop.
# show commands as they are run (debug only)
set -vx
# Find HeaderFiles (HF) matching F57DWHDR*.csv
for HF in F57DWHDR*.csv
# For each header file check 14 characters from 8th and ensure its >= 2013-01-01
do if [[ "${HF:8:14}" -ge 20130101000000 ]]
# Check that every line in HF ends with * (search for any rows not ending in *)
then grep -qv \*$ $HF
# Examine output of grep command
case $? in
# Records were found in file not ending in *
(1) echo FAIL2 $HF ;;
# File not found or syntax error
(2) echo FAIL1 $HF ;;
# All records ended with *: loop through line by line reading filename (FN) and expected count (CNT), ignoring the REST
(0) while read FN CNT REST
# check file exists and is 'readable'
do [ -r "$FN" ] \
# if not, fail and stop checking
|| { echo FAIL3 $FN; break; } \
# if readable, check line count of FN is equal to expected CNT
&& [ "$(wc -l < $FN)" -eq $CNT ] \
# if not equal, fail and stop checking
|| { echo FAIL4 $FN; break; } \
# if equal, show 'OK' and FN
&& echo OK $FN
done < $HF ;;
esac
# header filename is not correctly formed
else echo FAIL1 $HF
fi
done
---------- Post updated at 10:23 AM ---------- Previous update was at 08:25 AM ----------
An update ... have tested this a fair bit now and things are looking great, thanks again!
I had to take out the substring check for the date in the header filename as was getting 'bad subtitution' - from google perhaps I have an old version of ksh?
I switched the grep output cases (0) and (1) as 1 indicates no output (what we want).
One question: I realised my previous posted code with comments didn't work because it didn't have the \ to indicate its all still part of the same and/or ... putting \ on the end of all my comments didn't help either (guess they comment out the rest of the line) what's the best way to add comments to that section?
Thanks again, solution as it stands is below:
#! /usr/bin/ksh
# show commands as they are run (debug only)
#set -vx
# Check for HeaderFiles (HF) matching F57DWHDR*.CSV
if ls F57DWHDR*.CSV >/dev/null 2>&1; then
echo Header file found
else
echo No header files present
exit 1
fi
for HF in F57DWHDR*.CSV
# For each header file check it is readable
do if [[ -r $HF ]]
# Check that every line in HF ends with * (search for any rows not ending in *)
then grep -qv \*$ $HF
# Examine output of grep command
case $? in
# Records were found in file not ending in *
(0) echo $HF has records not ending in "*"
exit 1;;
# File not found or syntax error
(2) echo $HF not found or grep syntax error
exit 1;;
# All records ended with *: loop through line by line reading filename (FN) and expected count (CNT), ignoring the REST
(1) while read FN CNT REST
# check file exists and is 'readable'
do [ -r "$FN" ] \
|| { echo $FN is not present and readable; exit 1; } \
&& [ "$(wc -l < $FN)" -eq $CNT ] \
|| { echo $FN record count does not match header $CNT ; exit 1; } \
&& echo $FN has correct record count of $CNT
done < $HF ;;
esac
# header filename is not correctly formed
else
echo $HF is not readable
exit 1
fi
done
echo move files