Read multiple log files and create output file and put the result

OS : Linux 2.6.9-67 - Red Hat Enterprise Linux ES release 4

Looking for a script that reads the following log files that gets generated everynight between 2 - 5am

Master_App_20090717.log
Master_App1_20090717.log
Master_App2_20090717.log
Master_App3_20090717.log
Master_App4_20090717.log
Master_App5_20090717.log

One exception:

The log file Master_App_20090717.log gets data written into every 30 minutes in the 24hr period. This data is appended into the file and remains there. But each time the extract and transload start it start with the current time stamp in it begin line and end with current time stamp for a successful end or it could be Failed end.

In any case those entries remain in the file until we fix the issue and the next load starts and the results of next load will append to the same log file for that same day and will starts with the same begin line. So we need to make sure that script should not report previous failure that happend 35 mins ago and was reported by the same script 30 minutes ago. So we need to place a check that if it does find a failed end and it is more then 35 minutes ago then ignore it and continue reading the file and then report accordingly.

We will run this script in crontab

One script will run every morning after 5:00 am once all the load is finished or we will run 5 script seperately after a specific time to check the status

The other script will run 5 minutes pass every 30 minutes.

*************************************************************************************************
Success Message:

2009-02-25 07:24:24]Master_App start_time=[20090213 000000] end_time=[20090213 002959] ... begin

[Extract-2009-02-25 07:24:24]Execute Extract_App ... Begin 
[Extract-2009-02-25 07:24:24]Execute ex_nhv_104052_s ... 
[Extract-2009-02-25 07:26:14]Execute ex_nhv_104052_s ... Success 
[Extract-2009-02-25 07:26:14]Execute ex_nhv_104114_s ... 
[Extract-2009-02-25 07:28:06]Execute ex_nhv_104114_s ... Success 
[Extract-2009-02-25 07:28:07]Execute nhv_104094_s ... 
[Extract-2009-02-25 07:29:58]Execute nhv_104094_s ... Success 
[Extract-2009-02-25 07:29:58]Execute Extract_App ... Success

[Transload-2009-02-25 07:29:59]Execute Transload_App ... Begin 
[Transload-2009-02-25 07:29:59]Execute tl_nhv_ua_merge ... 
[Transload-2009-02-25 07:30:14]Execute tl_nhv_ua_merge ... Success 
[Transload-2009-02-25 07:30:16]Execute tl_nhv_104094_s ... 
[Transload-2009-02-25 07:39:56]Execute tl_nhv_104094_s ... Success 
[Transload-2009-02-25 07:39:57]Execute Transload_APP ... Success
Archiving data files ... Success

[2009-02-25 07:39:58]Master_App start_time=[20090213 000000] end_time=[20090213 002959] [Successful end]!

*************************************************************************************************
Failure Message:

[2009-02-25 08:55:55]Master_App start_time=[20090213 003000] end_time=[20090213 005959] ... begin

[Extract-2009-02-25 08:55:55]Execute Extract_App ... Begin 
[Extract-2009-02-25 08:55:55]Execute ex_nhv_104052_s ... 
[Extract-2009-02-25 08:57:43]Execute ex_nhv_104052_s ... Success 
[Extract-2009-02-25 08:57:43]Execute ex_nhv_104114_s ... 
[Extract-2009-02-25 08:59:33]Execute ex_nhv_104114_s ... Success 
[Extract-2009-02-25 08:59:33]Execute nhv_104094_s ... 
[Extract-2009-02-25 09:01:34]Execute nhv_104094_s ... Success 
[Extract-2009-02-25 09:01:34]Execute Extract_App ... Success

[Transload-2009-02-25 09:01:35]Execute Transload_eHealth_h ... Begin 
[Transload-2009-02-25 09:01:35]Execute tl_nhv_ua_merge ... 
[Transload-2009-02-25 09:01:51]Execute tl_nhv_ua_merge ... Success 
[Transload-2009-02-25 09:01:52]Execute tl_nhv_104094_s ... ERROR: ORA-03114: not connected to ORACLE DECLARE Archive_Files.prg bak exp_prdmmp_dev_20090123.dmp.gz......
....................................................ERROR at line 1: ORA-03113: end-of-file on communication channel SP2-0670: Internal number conversion failed Usage: { EXIT | QUIT } [ SUCCESS | FAILURE | WARNING | n | <variable> | :<bindvariable> ] [ COMMIT | ROLLBACK ] 
[Transload-2009-02-25 09:33:19]Execute tl_nhv_104094_s ... Fail 
[Transload-2009-02-25 09:33:20]Execute Transload_App ... Fail

[2009-02-25 09:33:20]Master_App start_time=[20090213 003000] end_time=[20090213 005959] [Failed end]!


*************************************************************************************************

Output file :

We could have 2 output files one for script that runs after 5 am and one for every 30 minute script

or

we could have 5 output files one for each log plus one for 30 minutes script output log so altogether 6 outfiles.

In the output file

First Delete the content of the file if any then

Leave blank if it is successful

Write "ETL Load Failed" if it finds any failed end.

---------- Post updated at 10:33 AM ---------- Previous update was at 10:32 AM ----------

Thank you in advance for your Help as i am desperately seeking a solution. Thank you and all the best

To keep the forums high quality for all users, please take the time to format your posts correctly.

First of all, use Code Tags when you post any code or data samples so others can easily read your code. You can easily do this by highlighting your code and then clicking on the # in the editing menu. (You can also type code tags

```text
 and 
```

by hand.)

Second, avoid adding color or different fonts and font size to your posts. Selective use of color to highlight a single word or phrase can be useful at times, but using color, in general, makes the forums harder to read, especially bright colors like red.

Third, be careful when you cut-and-paste, edit any odd characters and make sure all links are working property.

Thank You.

The UNIX and Linux Forums

Seems like you've simply copied the stuff from your project's technical document over here.

Since you are seeking a solution desperately, you must've tried something at your end. Instead of copying an entire specification, post just the relevant part of your scripts and ask *very* specific questions/doubts about the points where you got stuck.

tyler_durden