Working with files that are named with a timestamp

Shell Programming and Scripting
1

So I have a script that runs an ls on a given directory and looks for files with a timestamp that has the current hour in it and then does work with the files that it discovers

DATE=`date +'%y%m%d%H'`
HOSTIS=`hostname`
#EMAIL NOTIFICATION ALS
EMAIL=address@server.com
#

if [ $HOSTIS = 'serverhostname' ]; then
  TOHOST='destination_address@destination_server.com'
 else
#This is for the test server
 TOHOST='test_address@test_server.com'
fi

for i in `ls /transfer |grep $DATE`
do

   sftp  -vvv $TOHOST 2>&1 | tee /homedir/user/debug.out<<EFT
   cd /directory/file
      lcd /transfer
      put $i
EFT
   echo "SFTP Completed successfully."
   echo "SFTP Completed successfully." |mailx -s "SFTP completed successfully" $EMAIL
exit
done

This is then run as a cron at 58 after the hour. This has obvious implications if the user manages to generate data between hh:58:01 and HH:00:00. it was a small risk but a risk nonetheless. It only took them two weeks. So now I need to figure out how to search for $DATE -1 hour and run the cron based on that and it has me stumped...

any ideas?

2

Keep everything as is, except...
the 2nd line of the script add

sleep 180

which would make the time xx:01 but you are checking for things in the previous hour.

3

An interesting hack to do it, but wouldn't I need to adjust the cron run time as well? set it for 59 instead?

4

try assuming OSX:

date -v-[something you want] +'%y%m%d%H'

where [something you want] is a past time
-1H == one hour ago
-5M == 5 minutes ago.

So try -15M and run it at :05 after the hour.

A comment to make your life easier -- :

Those filename "timestamps" could easily be stale. ie., I started my process at 12:59.
It has 12 as the hour, because it was opened at 12:59, when open created the file.

But. It is now 13:02. My report process is still running! So, when you sftp, you are not going to get my whole file. It has not closed yet. To be safe you really should look at the mtime of the files.

5

No.
If the cron starts running at x:58
it will get the full data/time string at that moment
then wait three minutes (could be two, bu one more just to be sure)
then does everything else

Plus, if you start at x:59 perhaps (maybe?) the time will be x:00 by the time that command is returned. Why take a chance on it.

6

This is my fault, I always forget to mention distro. This is Solaris 10. As such it doesn't have the mmtime switch in find.

What it does have is -newer that compares the results against the mtime of the file called out after the -newer switch is set.

so

DATE=`date +'%y%m%d%H'`
NEWER=/users/userid/.newer
LOG=/users/userid//sftp.log
HOSTIS=`hostname`
#EMAIL NOTIFICATION ALS
#EMAIL=user@domain.com

if [ $HOSTIS = 'hostname' ]; then
  TOHOST='user@domain.com'
 else
#This is for the validation server
 TOHOST='user@domain.com'
fi


for i in `find /transfer -name '*.txt' -newer $NEWER -print `
do
   sftp  $TOHOST <<EFT
   cd /inbound/dir
      lcd /transfer
      put $i

EFT
   touch -a -m $NEWER;
   echo "SFTP Completed successfully." |mailx -s " SFTP completed successfully" $EMAIL
exit
done