I am trying to grep within a file to find and output certain parts of lines to other file(s). The output files need to have a dynamic file name based on a field in the main log.
The problem is that every line of the log is not the same, and often not even similar.
To explain further, the lines in the log look like:
2007-06-05 14:03:48,337 INFO External- PXgcGllGX1TMdFCXrKyc8GQTwvLlfQ6B9wYQLyGXTQpKX5yxW8FC!-1784053810!1181066628296|>> [HandleRequest] QService<?xml version="1.0" encoding="utf-8" standalone="yes"?>
2007-06-05 14:03:51,236 INFO External- PXgcGllGX1TMdFCXrKyc8GQTwvLlfQ6B9wYQLyGXTQpKX5yxW8FC!-1784053810!1181066628296|<< [HandleResponse] QService<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
2007-06-05 14:03:56,900 INFO External- |||>> [HandleRequest] QService<?xml version="1.0" encoding="utf-8" standalone="yes"?>
2007-06-05 14:03:58,492 INFO External- |||<< [HandleResponse] QService<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
2007-06-05 14:11:09,570 INFO External- |02-20070605-510669||>> [HandleRequest] LService<?xml version="1.0" encoding="utf-8" standalone="yes"?>
2007-06-05 14:11:12,752 INFO External- |02-20070605-510669||<< [HandleResponse] LService<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
2007-06-05 14:11:22,997 INFO External- |02-20070605-510669||>> [HandleRequest] AService<?xml version="1.0" encoding="utf-8" standalone="yes"?>
2007-06-05 14:11:38,191 INFO External- |02-20070605-510669||<< [HandleResponse] AService<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
What I want to do is ignore everything before the first pipe and take the '02-YYYYMMDD-XXXXXX' format between the pipes as the $FILENAME, then append everything after the XService (ie. <?xml to end of line) to the new file.
I would appreciate any suggestions, thanks in advance.
while read line
do
filename=`echo "$line" | sed 's/^.* |\(.*\)||\(.*\)/\1/'`
echo "filename is $filename"
echo "$line" | sed 's/^.*<?\(.*\)/<?\1>/' > $filename
done < source_file
As you said, untested, but so far good start.
Two main issues
First, it doesn't quite work. I neglected to mention that I only want to get what is after the HandleRequest for AService ([HandleRequest] AService).
It does make files in the current state, but selectively, and not the correct handle/service.
Second, is the millions of other files that are created named after random tags that are on their own lines. All the other garbage files all start with "<", what would the if statement in the brackets look like (if [ $filename != <* ] ?).
The script does work with my sample file but not every line has the same format. For example these lines are thrown in between the previous ones I posted.
It is interesting to see the different methods everyone has to tackle the same parse.
Ghostdog: I tried to add a bit to the if statement to only output lines that had '[HandleRequest] AService' in them (so that I would have only one xml, per 01/02-YYYYMMDD-XXXXXX), but I could not get it to work successfully.
Could you please post up this addition?
Jean-Pierre: Your awk script is very short and seems efficient, but with the addition of '&& /[HandleRequest] AService/', I cannot get any output. I have tried on many different servers, and while your original script creates the outputs, for some reason no matter what I try with the second one, I cannot get any output files. Do you have any suggestions? It is strange that it works for you but not for me at all. Is there an awk log or hidden verbosity I could enable to trace the activity? Also, how are you getting both of the AService lines output, one is a request and one is a reply, so one should definitely not pass your filters(?).
Thanks for all of your help ghostdog, Jean-Pierre, and matrixmadhan!
ghostdog: I have adjusted your second edit, and made it into a script, and it works with fantastic results on my very large log files.
How would I encorporate an if ( ! -f filename), as well as directory filing by date 01-YYYYMMDD-XXXXX into the corresponding YYYYMMDD directory) into the awk script?
Jean-Pierre: With your latest edit it works very well as well on the most complicated sample log I have provided, however my actual logs have more garbage in them that causes your script not to work. I have not been able to isolate what causes it to stop working. Also, when running with full very long xml code after the service search, the code is strangely truncated. I would like to send you some additional logs if you would not mind continuing to help me.
matrixmadhan: Your latest edit does work with my initial simple log, but also does stop functioning when it encounters more difficult log files, such as the one I posted in response to Jean-Pierre's early script. Additionally I only need to see the Request from Aservice.
Since this script will be run frequently on the same log file(s), I would like to:
-Add a check to make sure the script does not overwrite existing xml files (a check to see if the file exists, before it is written). I know how to do this as a shell script, but I am not sure how to do it in the awk script.
-While writing the new xml file (whose filename follow the format of 01-YYYYMMDD-XXXXXX - as read from the original log), move the xml into an appropriate directory (that can share the same YYYYMMDD format from the xml filename). If the filename does not follow the format (as above), then put it in an arbitrary saftey directory (any name).
#!/usr/bin/awk -f
# Awk script: extract3.awk
/HandleRequest.*OrderControlService/{
if ( b= match($0,"xml") ) {
xml = substr($0,b-2)
print $0
}
else {
next
}
n=split($0,line," ")
m=split(line[5],file,"|")
if ( file[2] ~ /^[0-9]/) {
filename=file[2]
if ((getline < filename) == -1) {
n=split(filename,array,"-")
print array[2]
cmd="mkdir -p " array[2]
print cmd
cmd | getline result
close(cmd)
print xml > array[2]"/"filename".xml"
}
else {
print filename " - File Exists"
}
}
}
Without the directory creation it works fine, however with this full version, when I run it I get:
It works on some shells, and gives me a mkdir missing operand error, but on the log server it just gives me the above error and does nothing.
What is wrong?
running
AIX 5.3, bash,
and I don't know what version of awk (there is no version flag?)
I fixed the original error I posted above, as my version of awk seems to have a problem with spaces in the script, so that is resolved for now.
But I have one more issue, which will hopefully be the last.
The way in which directories are created is by splitting the 01-YYYYMMDD-XXXXX by '-' then putting each piece in an array then creating a directory based on the second piece.
However there are some entries between the pipes that do not follow the format and have no '-' in them to give them a second piece.
For example there are some lines that look like:
2007-06-06 16:52:38,805 INFO External- |3676482||>> [HandleRequest] AService<?xml version="1.0" encoding="utf-8" standalone="yes"?>
I am not sure what would be easier, simply to put them in the same folder as the previous one that was created (rolling over the folder filename to the next loop), or to just put all all these seven digit standalone numbers that do not follow the array format into a fixed existing directory (by checking the size of array[2] before mkdir?).
it depends on what you are comfortable with. and you are on the right track. one way is to check the array[2], if empty then put to another defined directory..
