Grep from file modified one minute ago

Shell Programming and Scripting
1

Hello,

I have a list of files, an example below:

-rw-r--r--   1 smf_oper esg       910773 Jul  6 12:52 am1slc02_ACS_201607061242571_20346.cdr
-rw-r--r--   1 smf_oper esg       995838 Jul  6 12:52 am1slc01_ACS_201607061243125_19895.cdr
-rw-r--r--   1 smf_oper esg       557235 Jul  6 12:52 am1slc02_ACS_201607061243110_20309.cdr
-rw-r--r--   1 smf_oper esg       934318 Jul  6 12:52 am1slc02_ACS_201607061243164_20340.cdr
-rw-r--r--   1 smf_oper esg      1018777 Jul  6 12:52 am1slc02_ACS_201607061243160_20350.cdr
-rw-r--r--   1 smf_oper esg      1049209 Jul  6 12:52 am1slc02_ACS_201607061247501_20318.cdr
-rw-r--r--   1 smf_oper esg        15611 Jul  6 12:53 am1slc01_ACS_201607061243130_19939.cdr
-rw-r--r--   1 smf_oper esg      1048796 Jul  6 12:53 am1slc01_ACS_201607061247517_19953.cdr
-rw-r--r--   1 smf_oper esg      1048826 Jul  6 12:53 am1slc01_ACS_201607061247528_19963.cdr
-rw-r--r--   1 smf_oper esg      1049420 Jul  6 12:53 am1slc01_ACS_201607061246136_19909.cdr
-rw-r--r--   1 smf_oper esg         6911 Jul  6 12:53 am1slc01_ACS_201607061242428_19949.cdr
-rw-r--r--   1 smf_oper esg      1049219 Jul  6 12:53 am1slc01_ACS_201607061247364_19957.cdr

My aim is to try to grep from the files which were modified one minute ago, i.e. in the above example grep from the files which were modified at 12:52, separately since they are different file (example am1slc01* and am1slc02*)

I have tried to build a script below in which it managed to take the last files modified (not one minute ago yet) however the grep won't work, it seems the grep works only on the first file

#! /usr/bin/bash
 cd /IN/service_packages/SMS/cdr/received
 mydate=`date +"%H:%M"`
 myfiles=`ls -ltr am1slc01_* | grep $mydate | awk '{print$9}'`
 echo $myfiles
 echo Data:
r=`grep -s CALL_TYPE=VOICE-MO $myfiles | wc -l`
t=`echo "scale=10; ($r/60)*1000" | bc`
echo CAPmS-slc01 $t

When echoing the files they are not listed as when making a simple 'ls' command but near each other so I'm suspecting that's the reason why it won't work.

Any ideas?

2

You could try a simple while loop

ls -ltr am1slc01_* | grep $mydate | awk '{print$9}' |
while read fname
do
  r=`grep -s CALL_TYPE=VOICE-MO $fname | wc -l`
  t=`echo "scale=10; ($r/60)*1000" | bc`
  echo CAPmS-slc01 $t
done

FWIW:
Rather than messing around with all of those subprocesses you could do what you want with a single awk call.

1 Like
3

I support it also needs to be clarified as to what you mean by "one minute ago"

Is it:-

  • Within the last minute (ie. less than 60 seconds old)
  • Current minute (probably very few if the time is 12:53:01 but more if it's 12:52:59)
  • Exactly one minute (up to seconds/milliseconds)
  • At least one minute but less than two minutes
  • Files timestamped as the previous minute to now (ie. if I'm at 12:53, I want all 12:52 files, even if they are 90 seconds old
  • Something else?

There are probably several ways to get these, but we need to know exactly your definition of "one minute ago" first so we don't go off on a wild chase.

Kind regards,
Robin

4

Hi

The while loop seems to do the job, however I am now left with two or more outputs depending upon files found. For example when tested there were two files so grep was performed for both and output two values. Can these be summed up in the code?

 CAPmS-slc01 266.6666666000
CAPmS-slc01 16.6666666000

Also this loop still takes the last files but not one minute ago. To clarify i am trying to achieve what Robin listed below:

  • Files timestamped as the previous minute to now (ie. if I'm at 12:53, I want all 12:52 files, even if they are 90 seconds old

---------- Post updated 07-07-16 at 12:01 PM ---------- Previous update was 07-06-16 at 06:34 PM ----------

Hi,

just found a dirty solution for this. copy the files based on date:

mydate=`date +"%H:%M"`
 for i in `ls -lrt am1slc01_* | grep $mydate | awk '{print$9}'`; do cp $i /home/nms/scripts/CDRs/slc01; done
for i in `ls -lrt am1slc02_* | grep $mydate | awk '{print$9}'`; do cp $i /home/nms/scripts/CDRs/slc02; done

then do the required grep and delete files

Now the next step is to copy the files from one minute ago

5

You haven't said what you need that for, but i suppose you want to process "incoming" files somehow and want to prevent processing them twice.

In this case, wouldn't it be easier to move the files once they are processed? You wouldn't need to diddle with time stamps and complicated calculations and could adapt something like the following (sketch)

Two directories: /some/where/incoming and /some/where/processed , files go into "incoming"

while : ; do
     ls /some/where/incoming | while read FILE ; do
          process_file $FILE
          mv $FILE /some/where/processed
     done
     sleep 5
done

I hope this helps.

bakunin

6

Depending on what the process_file requirements are, you might gobble up the input to your while loop, especially if you have a while read loop. I'd suggest something more like:-

while :
do
   for FILE in /some/where/incoming/*
   do
      process_file "$FILE"
      mv "$FILE" /some/where/processed
   done
   sleep 5
done

If your files might have spaces in them, change the for loop to be:-

   for FILE in $(ls /some/where/incoming/*)

Does that help at all?

Robin