I want to append data to same .csv file.

Shell Programming and Scripting
1

I have a script which has to be scheduled to run 3 times a day.
My script picks the required fields from logfile and stores the data in a.csv file.
Sample data.

my logfile contain:

0097A,0374D,100903,1519,00000606191
0097A,C88RA,100903,0724,00000606105

So the output of first execution will be stored in a.csv file:

0097A,0374D,100903,1519,00000606191
0097A,C88RA,100903,0724,00000606105

Before second execution of script now logfile contain:

0097A,0374D,100903,1519,00000606191
0097A,C88RA,100903,0724,00000606105
0097A,C88RA,100903,0726,00000606106
0100A,0374A,100903,1358,00000355348
0115A,6652A,100903,1103,00000489132
0118A,6652A,100903,1301,00000339704

So my output should be appended to a.csv file keeping old data and appending new data to it with some line and time in between as below:
I want my output like this.

0097A,0374D,100903,1519,00000606191
0097A,C88RA,100903,0724,00000606105
-------------------2nd Phase $time--------------
0097A,C88RA,100903,0726,00000606106
0100A,0374A,100903,1358,00000355348
0115A,6652A,100903,1103,00000489132
0118A,6652A,100903,1301,00000339704

---------- Post updated at 05:55 AM ---------- Previous update was at 05:48 AM ----------

#!/bin/sh
time="`date '+%H%M%S'`"
date=`date +%y%m%d -d"1 day ago"`
inbound_dir=/vist/logfiles/to_solmis
mkdir -p /vist/sumit/in_ASN.$date.$time
cp `grep -il ST~856~ $inbound_dir/*$date*` /vist/sumit/in_ASN.$date.$time/
        for i in /vist/sumit/in_ASN.$date.$time/*
        do
            #cp `echo $i` /vis/sumit/inbound.$date
            tr -c '[A-Z] [0-9] [a-z] ~' '[ *]' <$i > b
            awk -F "DTM~" '{print $1}' b > c
            awk -F "~SH~" '{print $2}' c > d
            awk -F "~" '{print $1","$2","$3","$4","$11}' d >> a.csv.$date.$time
       done
            rm b
            rm c
            rm d
2

Not sure what you are trying to do with all those awk invocations in your shell script, but if you want to pick up the delta from an incrementing logfile then you could do something like this:

diff logfile csvfile | awk 'BEGIN{system("date '+%H%M%S'")} /^</{print $2}'

The diff command displays the delta records in the logfile with a preceding "<"; you filter those and redirect to your csvfile with the date on top.

tyler_durden

In case you want to embellish the delta header like so -

--- Phase 2 HH24MISS ---

then you'll have to figure out the last "phase" number first, increment it and then append it followed by the date and delta records.

This awk one-liner returns 2 if a phase isn't present in the csvfile, otherwise it returns the incremented phase number.

awk '/^--/{x=$3} END{print x==0?2:x+1}' csvfile

So you could assign it to a shell variable and pass that to your awk script like so -

PHASE=$(awk '/^--/{x=$3} END{print x==0?x+2:x+1}' csvfile)
diff logfile csvfile | awk -v P=$PHASE 'BEGIN{printf "--- Phase " P; system("printf \" %s ---\n\" `date '+%H%M%S'` ")} /^</{print $2}' >>csvfile

And finally, if the logfile doesn't exist then you simply copy csvfile to logfile. (This is the simplest part, and you should be able to do this on your own.)
For all other cases, the script posted above should work.

HTH,
tyler_durden

3

Actually my logfile contain a file,so using awk command I am picking the required data and storing in csvfile.
Logfile is not the same as csvfile.
csvfile contain some fields that are reuired.

So I cant differentiate logfile and csvfile.
As below mentioned in my script.

I want append data to a.csvfile

4

Sorry, the following line in my post:

And finally, if the logfile doesn't exist then you simply copy csvfile to logfile.

should read:

And finally, if the csvfile doesn't exist then you simply copy logfile to csvfile.

For the data you've provided, I believe diff should work.
If the data you've provided doesn't match your actual data (or the one that your script works on), then modify your example and show us exactly what you are trying to pick up from your logfile.

Finally, csvfile in my post should be considered a.csv (which was "a csv file" in your first post).

tyler_durden