I am attempting to parse an Audit Log from Cognos/TM1, selecting only Event IDs of "client" which are found on the "start-tag" record. These Logs are in a pseudo-XML format but not a true XML format. I want to FTP an Audit Log File from the Cognos server to our UNIX server. Then I want to look for "client" at position 28 in a given line, and upon finding that, write the current line and the next two lines after it as one output line. (We have Shell Scripts already that FTP data from Windows to UNIX, so I don't need assistance with that.)
These lines have CRLF (I think) after each one. When I view them in Windows-based editors (like TextPad), they are separate lines.
I'm a Unix script newbie. I've searched the Web for possible solutions but have not found any. I have found examples of merging two lines by patterns using sed and one using sed (I think) and a line range, but the sample had a hard-coded line range.
We run the AIX Flavor of UNIX at my company. We use KSH as the Shell Script most of the time.
Pseudo-code:
IF current-line-number line has "client" at position 28, write current-line-number concatenate current-line-number plus 1 concatenate current-line-number plus 2 FI
Welcome to the UNIX and Linux Forums.
Thank you for telling us what OS and shell you're using. That always makes it easier for the volunteers here who try to help answer your questions suggest solutions that will work in your environment.
Please provide a short sample input file and a corresponding output file showing what you want to be produced by your script for that sample input. (Please use CODE tags when showing us sample input, sample output, and code fragments.)
Input file sample (I put in "(crlf)" to show where crlf's are):
<Commit ts="20140812153817" client="AAAAAA">(crlf)
"144","AAAAAA","00.00.000.000","User 'AAAAAA' successfully logged in from address '00.00.000.000'."crlf
</Commit>crlf
<Commit ts="20140812160347" client="BBBBBB">(crlf)
"144","BBBBBB","00.0.000.00","User 'BBBBBB' successfully logged in from address '00.0.000.00'."(crlf)
</Commit>(crlf)
<Commit ts="20140812170838" client="CCCCCC">(crlf)
"144","CCCCCC","00.0.000.00","User 'CCCCCC' successfully logged in from address '00.0.000.00'."(crlf)
</Commit>(crlf)
Output file sample (again I put "(crlf)" to show where crlf's should be):
<Commit ts="20140812153817" client="AAAAAA">"144","AAAAAA","00.00.000.000","User 'AAAAAA' successfully logged in from address '00.00.000.000'."</Commit>(crlf)
<Commit ts="20140812160347" client="BBBBBB">"144","BBBBBB","00.0.000.00","User 'BBBBBB' successfully logged in from address '00.0.000.00'."</Commit>(crlf)
<Commit ts="20140812170838" client="CCCCCC">"144","CCCCCC","00.0.000.00","User 'CCCCCC' successfully logged in from address '00.0.000.00'."</Commit>(crlf)
Once in a single-record-per-event format, I can parse it using our ETL Tools.
should do what you want as long as the 1st line in every set of 3 adjacent lines in your input file contains "client" starting in column 29 (columns are numbered starting from 1; not 0; in awk . If you want to discard other types of records, this more complicated script should work:
awk '
cnt { out = out $0
if(--cnt == 0) {
gsub("\r", "", out)
print out "\r"
}
next
}
substr($0, 29, 6) == "client" {
cnt = 2
out = $0
}' inputfile > outuptfile
<Commit ts="20140812153817" client="AAAAAA">"144","AAAAAA","00.00.000.000","User 'AAAAAA' successfully logged in from address '00.00.000.000'."</Commit>crlf
<Commit ts="20140812160347" client="BBBBBB">"144","BBBBBB","00.0.000.00","User 'BBBBBB' successfully logged in from address '00.0.000.00'."</Commit>(crlf)
<Commit ts="20140812170838" client="CCCCCC">"144","CCCCCC","00.0.000.00","User 'CCCCCC' successfully logged in from address '00.0.000.00'."</Commit>(crlf)
Hi Ravinder,
The (crlf) (and in two places) crlf (without the parentheses) is FredAtArrow's notation indicating that the input and output lines have Windows style <carriage-return><line-feed> (where <line-feed> is a synonym for <newline> on Linux/UNIX systems) line terminators instead of just the Linux/UNIX style <newline> line terminators. The:
gsub(/[\r\n]/, "")
that processed every input line in Corona688's proposal is overkill removing both <carriage-return>s and <newline>s (I say overkill, because awk already strips out the <newline> characters before it hands us a line as long as we keep the default value for the awk RS variable).
The code I suggested strips out the carriage returns (the awk escape sequence \r in a string) from the accumulated set of three input lines just before adding back the desired <carriage-return> at the end of the line when printing the results:
gsub("\r", "", out)
print out "\r"
and, as we both know, as long as we are using the default awk ORS variable, the print command adds a <newline> when it prints a string.
Thank you everyone for your helpful posts! I used Don's code within a Kornshell Script (and I even got a loop of a List File of File Names to process multiple Raw files and parse them to Parsed Files to work!), and it did exactly what I needed it to do!
Here's the code I ran (Cognos_Audit_Log_File_List.lst file was pre-built earlier in the Script):
while read FILE
do
echo "$FILE" >> ${LOG_FILE}
awk '
cnt { out = out $0
if(--cnt == 0) {
gsub("\r", "", out)
print out "\r"
}
next
}
substr($0, 29, 6) == "client" {
cnt = 2
out = $0
}' ${SOURCE_DIR}/$FILE > ${TARGET_DIR}/$FILE
done < Cognos_Audit_Log_File_List.lst