Hi,
I have a log file which runs into 3 to 5 GB.
We store this typically for 6 months. When a new month starts we move the previous month into a 9 month back up log (file.9m) and delete the last month of the 9 month back up.
Iam using awk to find the data and cat to join the files like below
BEGIN { FS = "|"}
{ if (($5 == "09") && ($6 == "08")) print $0 >> "/file.1m";
}
END { print "script processed " NR}
in the above $5 is month and $6 is the year and the log file has pipe separated columns
cat /file.6m /new.log > /file.temp
mv /file.temp /new.log
the file.6m is the back up file when a new month starts and the new.log is the running log.
similarly I extract file.1m(last month) from file.6m and file.8m ( eight months without the last month from the file.9m) and join using cat as above.
This process though is NOT neat, is yielding the result.
I would like to know if there is any better of doing this. Since the file is really large sometimes i find the data jumbled up.
Appreciate your inputs.