While loop reading a record

Shell Programming and Scripting
1

Hi,

I am trying to create a while loop that will do the following:

INFILE= list of new records that need to be added after last previous record
while read record
do
find the last record processed
create list of new records output to a file
echo "$record">> $NEWFILE
done

Please advice. Thank You in advance

2

what do you mean by last previous record??

3

the INFILE will be a file of new records that need to be added to a file sorry.

4

I know I'm missing something but you want to add records in one file to another existing file - and add them at the end, right?

cat newrecordsfile >> oldrecordsfile

oldrecordsfile now has added all of the records in newrecordsfile

5

The thought process with this is you have the INFILE being updated every ten minutes lets say. So in the while loop you would read the record of the INFILE. Find the last record in the INFILE that was processed in another file called fileB. Create a new list called OUTFILE that captures the new files that need to be updated in fileB that was not processed after the last update of INFILE.

Hope this explains more.

6

I might be missing something, but...

tail -f newrecordsfile >> oldrecordsfile
7

that is only going to give me the last line. I need the last file processed in FileB to be in the starting point of where i left off in FileA. For example if i have FileA getting updated every ten minutes. FileA could start off with 1234 in the file that i send to FileB. So now FileB is 1234. But after 10 min FileA is now 123456. How do i send 456 to the end of FileB?

8

I typically take one of two approaches

  1. capture the current line count to a variable in a script that remains in memory
  2. capture the current line count to a file so you can read it in the next time you need it
    The next time you read fileA use tail +$previous_line_count >> fileB
9

this is what i have right now:
INFILE=input.txt
OUTFILE=output.txt

LINECOUNT= wc -l < $INFILE
OUTCOUNT=$LINCOUNT
tail -f $INFILE + $OUTCOUNT >> $OUTFILE

Right now with the input file being the following:
1
2
3
4
5
6
7
8
9

with running this
I am only getting:
5
6
7
8
9
in the output.txt

Also how do i put this in while loop?

The idea is the input.txt file is going to be updated regularly.So with the original input being:
1
2
3
4
5
6
7
8
9
If the file is update to put:
10
12
13
at the end file how do i add that to the end of the output.txt file in a while loop

10

Here is an option that will allow the script to run until a time in the future

INFILE=input.txt
OUTFILE=output.txt
PREV_LINECOUNT=0
CURRENT_TIME=`date +%H%M`

# # time format HHMM
EXIT_TIME="$1"

# # keep gather data until you reach exit time
until (( $CURRENT_TIME >= $EXIT_TIME )); do
  LINECOUNT=`wc -l < $INFILE`
  CURRENT_TIME=`date +%H%M`
  tail +${PREV_LINECOUNT} $INFILE >> $OUTFILE
  PREV_LINECOUNT=`expr $LINECOUNT + 1`
  # # wait 1 min and get data again
  sleep 60
done

exit 0

If you want it to run until you tell it to stop then replace the until line with a while loop that will require you to use ctrl+c to exit the script

while true; do