Polling file

Shell Programming and Scripting
1

HI
I need some help on this below one
Have summuary file coming on daily basis with list of file names and count ,if the summuary file not exist ,pool it for every 5 mins till it arrives .Once arrived remove first and last line from the file and check all the files in respective director all are exist it,If it exists compare count with file against file exist in OS ,if it matches for all the files then move it some other server .Can any help from any one ?

Regards,
MM

---------- Post updated at 01:44 AM ---------- Previous update was at 01:18 AM ----------

Take all the files from this test file and check whether all are exist ,if all are exist then move to some other direcotry

Example
summary.txt

 PASSIVELBS_xxx_ 1 
 PASSIVELBS_xxx_ 2 
 PASSIVELBS_xxx_ 3
2

Hi, what have you tried so far and where are you stuck?

3

Hi
want to check ,grep and sed cannot use single like ,i want to find **** lines in file,if exist only remove those lines .When i am doing the below check throwing error

CHECK_FILE=`egrep -i '******' ${BASE_PATH}/${FILENAME}|sed -i '1d;$d'`

sed: no input files

4 
if [ -f  ${BASE_PATH}/${FILENAME} ] ; then
    awk ' index($1, "*")==1 {next} 
            {print} ' ${BASE_PATH}/${FILENAME}  > tmp.tmp
    mv tmp.tmp ${BASE_PATH}/${FILENAME}
fi

This deletes the lines you want deleted. Drop the mv command if you want to keep the file as it was. Your original code does not make sense to me. The CHECK_FILE variable will always be "empty".

5

Thanks jim,its working perfect.I need one more conditions to check
Let say summary.txt having 3 files but when checking only 2 files are available in that location ,ensure that 3 files should be exist before proceeding to another step .

summary.txt
========
PASSIVELBS_xxx_ 1 
PASSIVELBS_xxx_ 2
PASSIVELBS_xxx_ 3
6

PLEASE DON'T edit posts after people have replied to it, pulling the rug from under their feet (and here: yours as well) and rendering their efforts useless!

With your modifications of post#1, jim mcnamara's proposal becomes a pointless no-op.
And, BTW, grep ping for a pattern like '****' as you do in post#3 won't work as expected because * is a regex multiplier (zero or more times) character. To remove starred and blank lines from your input, try

grep -v '\*\|^ \+$' file

The error in post#3 is due to you wanting sed to "edit-in-place" stdin from a pipe.

Re your new request: do you just want a file count of three, or should exactly those three listed files exist? If the former: are there always exactly two header lines to be skipped?

3 Likes
7

Hi ,
Below code is working fine,exit from loop if not matching .I need to be loop forever untill match both files count .Any help would be appereciated

summary=`cat ${BASE_PATH}/${FILENAME|wc -l`
check=`ls -l PASSIVELBS_xxx_*|wc -l`
while [ $summary != $check ];do 
echo "Summary files are $summary"
echo "Checksum files are $check"
echo "Files are not matching with checksum"
exit 1
8

Give us a break! The above code cannot possibly be "working fine".

  1. ${FILENAME is a syntax error. (Where is the closing } ?)
  2. BASE_PATH is not defined!
  3. FILENAME is not defined!
  4. while [ $summary != $check ];do has no matching done ! (Another syntax error.)
9

Sorry for not giving in details ,

BASE_PATH=/home/tmp
FILENAME=summary.txt==>3 files 
These are exist in current location and wait for 3rd file to arrive  

PASSIVELBS_xxx_1
PASSIVELBS_xxx_2

summary=`cat ${BASE_PATH}/${FILENAME}|wc -l`
check=`ls -l PASSIVELBS_xxx_*|wc -l`
while [ $summary != $check ];do 
echo "Summary files are $summary"
echo "Checksum files are $check"
echo "Files are not matching with checksum"
exit 1
10

On top of what Don Cragun already said, even your new "detailed" code suffers from

  • no closing done
  • entering an infinite loop as the comparison will never change

Without decent answers to my questions in post#6, assuming

  • one of the possible scenarios
  • a recent bash
  • file count will change and not the summary.txt file
  • file count exceeding summary line count is unacceptable or will never occur

, this might work for you:

SUM=$(wc -l < summary.txt)
while (( SUM - 2 - $(ls PASSIVE* | wc -w) ))
  do    echo NO
        sleep 1
  done
11

Please oh please, read your code again!
You are here to obtain help and this is the second time this code won't work.
Don has already told you many faults now you have others...

BASE_PATH=/home/tmp
FILENAME=summary.txt==>3 files 
These are exist in current location and wait for 3rd file to arrive  

PASSIVELBS_xxx_1
PASSIVELBS_xxx_2

summary=`cat ${BASE_PATH}/${FILENAME}|wc -l`
check=`ls -l PASSIVELBS_xxx_*|wc -l`
while [ $summary != $check ];do 
echo "Summary files are $summary"
echo "Checksum files are $check"
echo "Files are not matching with checksum"
exit 1

Longhand, OSX 10.7.5, default bash terminal...

Last login: Sun Mar 12 10:12:11 on ttys000
AMIGA:amiga~> cd Desktop/Text
AMIGA:amiga~/Desktop/Text> chmod 755 junk.txt
AMIGA:amiga~/Desktop/Text> cat junk.txt
BASE_PATH=/home/tmp
FILENAME=summary.txt==>3 files 
These are exist in current location and wait for 3rd file to arrive  

PASSIVELBS_xxx_1
PASSIVELBS_xxx_2

summary=`cat ${BASE_PATH}/${FILENAME}|wc -l`
check=`ls -l PASSIVELBS_xxx_*|wc -l`
while [ $summary != $check ];do 
echo "Summary files are $summary"
echo "Checksum files are $check"
echo "Files are not matching with checksum"
exit 1
AMIGA:amiga~/Desktop/Text> 
AMIGA:amiga~/Desktop/Text> 
AMIGA:amiga~/Desktop/Text> ./junk.txt
./junk.txt: line 2: files: command not found
./junk.txt: line 3: These: command not found
./junk.txt: line 5: PASSIVELBS_xxx_1: command not found
./junk.txt: line 6: PASSIVELBS_xxx_2: command not found
cat: /home/tmp/: No such file or directory
ls: PASSIVELBS_xxx_*: No such file or directory
./junk.txt: line 15: syntax error: unexpected end of file
AMIGA:amiga~/Desktop/Text> _

As there is no line 15 then you still have an error that Don has already told you about!
Please be accurate as it helps us to help you.

12

FWIW - Is there some logical reason why you cannot directly observe the 3 files you want instead of all of this other indirect logic.

This has a timed loop that returns an error after one hour of waiting.
It also polls the files once every 10 seconds, and emails okay or failure

#!/bin/bash
# file: poll.sh
cnt=0
now=$( date +%s )   # time in epoch seconds
when=$(( $now + 86400 ))  # we will only wait 1 hour then produce an error

while [ $cnt -le 3 ]
do
     if [ $( date +%s ) -gt $when ]; then
         echo 'Timed out: failure to find files' | 
              mailx -s 'poll.sh failed'  mohan@somewhere.com
         exit 1
     fi
     cnt=`ls -1 PASSIVE_XXX_? | wc -l`   # cnt will be at least 3 in order to end loop
     sleep 10   # wait 10 seconds before trying again.
done
echo 'All files found' | mailx -s 'poll.sh okay'  mohan@somewhere.com
exit 0

usage:

/path/to/poll.sh

This is somewhat verbose but it shows exactly what you need to do. Ignoring the likelihood that the filenames (PASSIVE*) are not as you presented them.

13

Using a recent bash 's features, you can test if exactly the named files are present:

mapfile -ts2 FNS <summary.txt
while [ ${MISS:-1} -eq 1 ]
  do    MISS=0
        for I in ${!FNS[@]}
          do    [ -e ${FNS[$I]} ] || { MISS=1; sleep 1; }
          done
  done

This will create an array with the file names expected, and then loop until ALL files exist; if any one is missing, it will wait some time.