Need to process files created an hour ago

Shell Programming and Scripting
1

Hello all,

I would like to ask for an advice on how to deal with the following scenario.

Every now and then, our ERP system creates an interface text file with the following file format - XORD????.DLD where ???? is a sequence number. We can have 1 or more XORD files created in an hour. Now, I would like to create a script that is run every hour to check XORD* created for the last hour and then verify its contents. How do I approach this using scripting (ksh). The first thing that comes into my mind is to save the result of `ls -lt XORD` to a text file and then sort it but not sure how to compare the time stamp with the current time. Thank you very much

Sample listing of files, say if I run my script now, I wont get anything......

-rw-rw-rw- 1 amcgraw dba 5609 Jun 14 07:17 XORD9306.DLD
-rw-rw-rw- 1 root sys 172859 Jun 14 00:22 XORD9305.DLD
-rw-rw-rw- 1 ldowning dba 7287 Jun 13 17:24 XORD9304.DLD
-rw-rw-rw- 1 root sys 39985 Jun 13 17:05 XORD9303.DLD
-rw-rw-rw- 1 swelch dba 1427 Jun 13 16:20 XORD9302.DLD
-rw-rw-rw- 1 athomas dba 1207 Jun 13 16:09 XORD9301.DLD
-rw-rw-rw- 1 shicks dba 73610 Jun 13 16:07 XORD9300.DLD
-rw-rw-rw- 1 kscott dba 9860 Jun 13 15:52 XORD9299.DLD
-rw-rw-rw- 1 jrobinso dba 2404 Jun 13 15:28 XORD9298.DLD
-rw-rw-rw- 1 dburke dba 2436 Jun 13 14:48 XORD9297.DLD
-rw-rw-rw- 1 srainwat dba 1936 Jun 13 14:26 XORD9296.DLD
-rw-rw-rw- 1 kscott dba 8829 Jun 13 14:18 XORD9295.DLD
-rw-rw-rw- 1 mgwaltne dba 2532 Jun 13 13:57 XORD9294.DLD
-rw-rw-rw- 1 nthackre dba 2754 Jun 13 13:47 XORD9293.DLD
-rw-rw-rw- 1 dburke dba 2186 Jun 13 13:29 XORD9292.DLD
-rw-rw-rw- 1 shicks dba 6543 Jun 13 13:27 XORD9291.DLD
-rw-rw-rw- 1 dburke dba 62840 Jun 13 12:34 XORD9290.DLD
-rw-rw-rw- 1 shicks dba 75595 Jun 13 11:55 XORD9289.DLD

Joseph

2

Hi Joseph

Can the file name change once you have processed it ? if so, just rename the file to a name that has a extension of the time inclusive of second it was run.

J

3

Thanks J.

The file name cannot change after processing that is why i have to depend on the `ls` information. :frowning:

4

Any help is much appreciated...please. :slight_smile:

5

On each run `touch timestamp_file`
Then you can `find -newer timestamp_file`

You haven't given enough info but here is some
bash to process each new file seperately,
assuming all the files are in 1 directory:

#!/bin/sh

WORK_DIR=/var/reports
TSF=/tmp/batch_process.timestamp

[ ! -e $TSF ] && touch -r $WORK_DIR $TSF

find $WORK_DIR -newer $TSF -type f -maxdepth 1 |
while read REPORT; do
    echo processing $REPORT
done

touch $TSF
6

Thanks!

The files are only created in one directory and all files starts with XORD. These files are created anytime within the hour and every top of the hour, I would like to check all XORD files created the last hour on the time my script run (thru cron).

Say if my script runs at 3:00PM, I would like to get the list of XORD files created between 2:00-2:59PM and then put it in a variable so I can check each file if the contents are correct.

I tried using the find command using option -ctime, atime and mtime but this is by the day and sometimes the results is not consistent (or maybe I am not using it or understand it correctly...)

THanks!

Joseph

7

The timestamp option should work, and be more general.
The hourly period is just determined/configured in cron.
If you do want to have problematic edge cases where
you may miss files or process files twice, then use the following:

find $WORK_DIR -mmin -60 -type f -name "XORD*" -maxdepth 1

8

Thanks!

I will try the mmin option.

I tried the maxdepth option but it gave me an error. I am using hp-ux 11 ksh shell.

I have another question, what does the following line mean?

[ ! -e $TSF ] && touch -r $WORK_DIR $TSF

Thanks!

Joseph

9

Note I suggest you reconsider the timestamp
option, as otherwise crond and -mmin -60 are
just racind against each other.

The code snippet means:
[ If the timestamp file doesn't exist ] create it with the timestamp of the directory.
I.E. on the first run, all existing files in the directory will be processed.

10

Thank yiou very much!

Joseph