Help me with daily monitoring script

RaghavendraT · June 21, 2013, 5:03am

find /usr/IBM/FileNet/BulkUploaderScript/$i/Log/SuccessLog \( -newer range_start -a \! -newer range_end \)
  -exec wc -l {} \; |awk '{total+=$1-1} END {print total}' >>$LOGFILE

I need to print time stamp of the files int the paticular period .Please modify the above script so that i get timestamp of the files:b:

rbatte1 · June 21, 2013, 5:14am

I would start by changing wc -l into ls -l and see if that helps.

What output are you trying to achieve?

Robin
Liverpool/Blackburn
UK

RaghavendraT · June 21, 2013, 5:23am

Hi ,

I need number of lines in the file and time stamp of the file .
Example: I will pass start date(20130607) and end date(20130608) and i am hard coding time in my code as it is fixed in my daily monitoring task(5:30 AM to 5:30AM). Files will be uploaded in this Path "/usr/IBM/FileNet/BulkUploaderScript/Log/SuccessLog ". so I need filename with time and number of lines in the file those uploaded in that paticular time .Now i am getting number of lines but not timestamp and name. please tell me a way how to modify the above code.

rbatte1 · June 21, 2013, 5:41am

You may need to write a loop rather than try to do it in one find command.

Assuming you are finding the right files, rather than -exec .... try:-

find /usr/IBM/FileNet/BulkUploaderScript/$i/Log/SuccessLog \
   \( -newer range_start -a \! -newer range_end \) \
   | while read file
do
   lines=`wc -l $file`
   ls -l $file | read w x y z d1 d2 d3 rest
   echo "File $file has $lines lines and a date of $d1 $d2 $d3"
done

.... and adjust it to your needs.

You may need to check that the date is in the correct fields. Of course, after a few months, most systems switch the format from time day month to day month year, so that might need consideration.

Does that give you a starting point?

Robin

RaghavendraT · June 21, 2013, 6:40am

Hi ,
I am getting an error like this

wc: cannot open /usr/IBM/FileNet/BulkUploaderScript/LogsFolder/Log/SuccessLog/CSVSuccessLogHIPAACERTS7-6-2013
wc: cannot open 15-32-18.csv
/usr/IBM/FileNet/BulkUploaderScript/LogsFolder/Log/SuccessLog/CSVSuccessLogHIPAACERTS7-6-2013 not found
15-32-18.csv not found.

And i am not able to extract timestamp of the file . please modify this if u can.

---------- Post updated at 05:40 AM ---------- Previous update was at 05:40 AM ----------

But the File present in location

rbatte1 · June 21, 2013, 11:35am

It;s probably lack of permissions for the "cannot open" messages. If you run id and look at the permissions of the file, can you read it? Are you in the right group?

Could be someone/thing deleting files as you are running for the "not found" message. not sure what you can do about that really. There will be a read of the directory first to get the list into memory, then in the loop it will query the details.

Robin

RaghavendraT · June 26, 2013, 8:15am

Thanks for the code, but i am still facing

wc: cannot open /usr/IBM/FileNet/BulkUploaderScript/LogsFolder/Log/SuccessLog/CSVSuccessLogHIPAACERTS7-6-2013
wc: cannot open 15-32-18.csv

This the file name "CSVSuccessLogHIPAACERTS7-6-2013 15-32-18.csv" .One thing i noticed it has space between date and timestamp.It has all permissions(777).

Please see into this .I guess it is with WC command.

Just_Ice · June 26, 2013, 9:33am

the while loop is treating the file name with the space as 2 files ... you may want to check for the space in the file name from your find output prior to sending it to the loop (see code below) ... use sed or awk or whatever is quicker to replace the spaces with a separator character in place of cat|tr as you see fit ... sample could be optimized further ...

find /usr/IBM/FileNet/BulkUploaderScript/$i/Log/SuccessLog \
   \( -newer range_start -a \! -newer range_end \) > /tmp/listfile

cat /tmp/listfile | tr " " "_" > /tmp/listfile.1

mv /tmp/listfile.1 /tmp/listfile

for file in $(< /tmp/listfile)
do
   lines=`wc -l $file`
   ls -l $file | read w x y z d1 d2 d3 rest
   echo "File $file has $lines lines and a date of $d1 $d2 $d3"
done

rbatte1 · June 26, 2013, 9:48am

Surely that would then try to find the wrong file name again? If the file is called "abc def" and you run against it, will it not try to list "abc_def" which doesn't exist?

Would some tweak like this help?

find /usr/IBM/FileNet/BulkUploaderScript/$i/Log/SuccessLog \
   \( -newer range_start -a \! -newer range_end \) \
   while read file
   do
      lines=`wc -l "$file"`
      ls -l "$file" | read w x y z d1 d2 d3 rest
      echo "File \"$file\" has $lines lines and a date of $d1 $d2 $d3"
   done

There's also now no files getting written and re-written then read back in again.

Robin

Just_Ice · June 26, 2013, 9:59am

assuming find command output is in file.txt, as a demonstration on how your find-while loop works ...

file.txt

abc def
ghi.klm
nopqrst

run ..

for file in $(< file.txt)
do
    echo "$file"
do

please check output and let us know what you see ...

rbatte1 · June 26, 2013, 10:35am

Yes Just Ice, that's a problem, however that is why my suggestion in post #9 has a while read file loop rather than for file in loop.

I get the screen scrape:-

$ cat file.txt
abc def
ghi.klm
nopqrst
$ for file in $(< file.txt)
> do
>     echo "$file"
> done
abc
def
ghi.klm
nopqrst
$ while read file
> do
>     echo "$file"
> done <file.txt
abc def
ghi.klm
nopqrst
$

Does this get you towards solving the problem, RaghavendraT

Robin

RaghavendraT · June 27, 2013, 2:43am

Thanks alot now it is able to read the file with spaces.But { ls -l "$file" | read w x y z d1 d2 d3 rest } this is not working i mean it is not printing the date and time stamp of file .

rbatte1 · June 27, 2013, 5:10am

Can you post the code that you have now, a simple ls and the output you are seeing please.

Thanks,
Robin

RaghavendraT · June 27, 2013, 5:40am

Hi,
This is the ouput after execuition
{File /usr/IBM/FileNet/BulkUploaderScript/LogsFolder/Log/SuccessLog/CSVSuccessLogHIPAACERTS7-6-2013 15-32-18.csv has 4 /usr/IBM/FileNet/BulkUploaderScript/LogsFolder/Log/SuccessLog/CSVSuccessLogHIPAACERTS7-6-2013 15-32-18.csv-1 lines and a date of}

I guess we should some change in this line
"ls -l "$file" | read w x y z d1 d2 d3 rest" and we need the time stamp of the file when it got created.

rbatte1 · June 27, 2013, 5:48am

Two things:-

Where have the quotes round the file name gone / why are there { and } around the output?
We don't think you can get the create time, just the modification time.

Can you post the code as you have it?

Thanks,
Robin

RaghavendraT · June 27, 2013, 5:56am

Hi Robin,

Code remains same as we disscussed i.e

find /usr/IBM/FileNet/BulkUploaderScript/$i/Log/SuccessLog \
\( -newer range_start -a \! -newer range_end \) \
| while read file
do

lines=`wc -l "$file"`
ls -l "$file" | read w x y z d1 d2 d3 rest
echo "File $file has $lines-1 lines and a date of $d1 $d2 $d3"
done

Ouptut :

File /usr/IBM/FileNet/BulkUploaderScript/LogsFolder/Log/SuccessLog/CSVSuccessLogHIPAACERTS7-6-2013 15-32-18.csv has        4 /usr/IBM/FileNet/BulkUploaderScript/LogsFolder/Log/SuccessLog/CSVSuccessLogHIPAACERTS7-6-2013 15-32-18.csv-1 lines and a date of

I specified {} to tell it is output,don't take that into consideration .Last modification time is also meets the my Dm requirement.

Thanks
Raghav

RaghavendraT · July 10, 2013, 3:22am

Hi,

do
     lines=`wc -l "$file"`
   ls -l "$file" | read w x y z d1 d2 d3 rest
   echo "File $file has $lines  lines and a date of $d1 $d2 $d3"
done

OUTPUT:

File /usr/IBM/FileNet/BulkUploaderScript/LogsFolder/Log/SuccessLog/CSVSuccessLog                                                                                        HIPAACERTS7-6-2013 15-32-18.csv has        4 /usr/IBM/FileNet/BulkUploaderScript                                                                                        /LogsFolder/Log/SuccessLog/CSVSuccessLogHIPAACERTS7-6-2013 15-32-18.csv lines                                           and a date of

Could u please modify this so that i can get last modification date and time of the file .

rbatte1 · July 10, 2013, 8:09am

Sorry, I have been away for a while.

So,

do
     lines=`wc -l "$file"`
   ls -l "$file" | read w x y z d1 d2 d3 rest
   echo "File $file has $lines  lines and a date of $d1 $d2 $d3"
done

You say it gives the output:-

File /usr/IBM/FileNet/BulkUploaderScript/LogsFolder/Log/SuccessLog/CSVSuccessLog

.... but I don't see the literal text that should be there. The lines and a date of bit. I'm confused.

Can you post a section of output from a simple ls -l any_file so I can check the way it will be presented.

Thanks,
Robin

RaghavendraT · July 10, 2013, 8:16am

Output of ls -1 :

File /usr/IBM/FileNet/BulkUploaderScript/LogsFolder/Log/SuccessLog/CSVSuccessLogHIPAACERTS7-6-2013 15-32-18.csv has    4 
/usr/IBM/FileNet/BulkUploaderScript/LogsFolder/Log/SuccessLog/CSVSuccessLogHIPAACERTS7-6-2013 15-32-18.csv lines and a date of

I guess it the problem is with the read line so in print statement it is not printing d1 ,d2,d3.

ls -l "$file" | read w x y z d1 d2 d3 rest 
echo "File $file has $lines-1 lines and a date of $d1 $d2 $d3"

Just_Ice · July 10, 2013, 9:15am

your $lines output comes out as 4 filename and not what you intended as just 4 ... your ls -l file | read line is not setting any variables either ... you may want to recheck the appropriate lines on the command line before putting them into the script ...

lines=`wc -l "$file"`   # <-- may be better as lines=`wc -l < $file`

ls -l "$file" | read w x y z d1 d2 d3  # <-- this line is not setting variables