script for taking the stats from a file

i have file which contains data like this every day.i need to pull up a report for counting the 203's in that file for each subscriber id.there are around 200 subscriber id's. all ths Y's which i have written in the script are the subscriber id's.could some one give me an idea as to how do it in a better way.

more 09031000010202.TL
203,0903100000042C,710,29,3,
207,0903100000042C,670,133,1
207,0903100000042C,26,9,7,09
207,0903100000042C,42,61,1,0
207,0903100000052C,710,29,7,
207,0903100000052C,50,7,7,09
207,0903100000052C,218,22,1,
207,0903100000052C,36,25,7,0
203,0903100000052C,720,24,3,
203,0903100000052C,36,79,3,0

Script:

#!/bin/sh
set -x
thisday =`date +%y%m%d00010202.TLG`

for Y in 2 144 9 129 16 37 42 45 87 104 53 40 62 63 51 158 18 19 20 21 50 67 82 116 64 100 141 168 170 171 49 84 85 35 34 70 61 69 8 26 27 28 78 79 32 33 10 11 130 25
36 47 57 200 173 6 46 48 55 56 72 118 22 13 110 109 203 29 80 12 660 222 696 664 666 669 670 671 672 674 675 676 677 39 207 208 209 210 211 212 680 681 682 43 683 59 6
84 103 659 205 91 689 690 692 30 215 216 217 218 697 698 700 52 174 705 706 707 220 708 710 711 712 713 714 715 716 717 718 719 720 721 722 23 31 724 219 77 88 17 66 7
30 83 68 120 96 5 667 678 679 685 186 15 726 733 728 731 732 734 221 736 94 95 97 99 41 74 65 105 738 98 38 73 7 4 739 44 740 112 113 114 14 3 1 24 54 81 58;
do
echo "$Y 203's = ";`cat $thisday | grep ^203 | grep "2C,$Y, | wc -l`; done

That's complicated , this should work:

awk -F, 'NR==FNR{_[$0];next}$1==X && $3 in _{a++}END{print a}' X="203" subscriber.list `date +%y%m%d`00010202.TLG

should i put this into a subscriber list file

2 144 9 129 16 37 42 45 87 104 53 40 62 63 51 158 18 19 20 21 50 67 82 116 64 100 141 168 170 171 49 84 85 35 34 70 61 69 8 26 27 28 78 79 32 33 10 11 130 25
36 47 57 200 173 6 46 48 55 56 72 118 22 13 110 109 203 29 80 12 660 222 696 664 666 669 670 671 672 674 675 676 677 39 207 208 209 210 211 212 680 681 682 43 683 59 6
84 103 659 205 91 689 690 692 30 215 216 217 218 697 698 700 52 174 705 706 707 220 708 710 711 712 713 714 715 716 717 718 719 720 721 722 23 31 724 219 77 88 17 66 7
30 83 68 120 96 5 667 678 679 685 186 15 726 733 728 731 732 734 221 736 94 95 97 99 41 74 65 105 738 98 38 73 7 4 739 44 740 112 113 114 14 3 1 24 54 81 58

You should put each subscriber id on a separate line

2
144
9
etc.

ins@ARTMGA01> awk -F, 'NR==FNR{_[$0];next}$1==X && $3 in _{a++}END{print a}' X="203" a.out `date +%y%m%d`00010202.TLG
333868
ins@ARTMGA01>

i get this when i run the above query i am not getting the count of 203's for each subscriber id

subscriber id is the list above

$ nawk -F, 'NR==FNR{_[$0]=0;next}$1==x && _[$3]!=""{a[$3]++}END{for(i in a){print i,a}}' x=203 subscriber.list `date +%y%m%d`00010202.TLG
720 1
710 1
36 1

this works but how do we pull it for the whole month of february and then the ouput doesntcome for subscriber id's which arenot there thats if a 203 is not in the file it doesnt show for
2 0
144 0

one more thing can we put in an echo statement there like this echo "$i 203's"

You are not very clear explaining what you want. Your original request is to only include lines starting with 203. If you wish get all records irrespective of whether the subscriber has used or not used, then you can do this:

nawk -F, 'NR==FNR&&$1==203{_[$3]++;next}NF<2{b[$0]++}END{for(i in b){_=="" && _=0; print i,_}}'  datafile subscriber.list

My question was only how do i pull out stats for the whole month.
also it gives the number of 203's only for the subcribers ids which are there in the log file. even if a 203 is not there in the traffic log file for the subscriber id 2 or any other subscriber id it should show a zero.

like for example its doesnt give
2 0
7 0
if there are no 203's.

also can i put an echo statement there.echo "$Y 203's = "

where $Y is the subcriber id

grep ^203 testS | cut -d, -f3 | sort | uniq -c | while read line; do
cnt=$(echo $line | cut -d" " -f1)
acct=$(echo $line | cut -d" " -f2)
echo "Account $acct had $cnt 203's"
done

This is not as elegant as awk but you don't have to keep a list of accounts. You can change the "echo" line to contain your date variables etc..

thanks you testS and where do i put the log file in there.
-rw-r--r-- 1 ins users 203905773 Dec 15 00:00 08121400010202.TLG
-rw-r--r-- 1 ins users 236227823 Dec 16 00:00 08121500010202.TLG
-rw-r--r-- 1 ins users 243940118 Dec 17 00:00 08121600010202.TLG
-rw-r--r-- 1 ins users 237871341 Dec 18 00:00 08121700010202.TLG
-rw-r--r-- 1 ins users 250888222 Dec 19 00:00 08121800010202.TLG
-rw-r--r-- 1 ins users 224493872 Dec 20 00:00 08121900010202.TLG
-rw-r--r-- 1 ins users 218388629 Dec 21 00:00 08122000010202.TLG
-rw-r--r-- 1 ins users 186587338 Dec 22 00:00 08122100010202.TLG
-rw-r--r-- 1 ins users 212704014 Dec 23 00:00 08122200010202.TLG
-rw-r--r-- 1 ins users 223440201 Dec 24 00:00 08122300010202.TLG
-rw-r--r-- 1 ins users 205148228 Dec 25 00:00 08122400010202.TLG
-rw-r--r-- 1 ins users 190723897 Dec 26 00:00 08122500010202.TLG
-rw-r--r-- 1 ins users 196810206 Dec 27 00:00 08122600010202.TLG
-rw-r--r-- 1 ins users 207268885 Dec 28 00:00 08122700010202.TLG
-rw-r--r-- 1 ins users 192394546 Dec 29 00:00 08122800010202.TLG
-rw-r--r-- 1 ins users 213904424 Dec 30 00:00 08122900010202.TLG
-rw-r--r-- 1 ins users 227873419 Dec 31 00:00 08123000010202.TLG
-rw-r--r-- 1 ins users 261625589 Jan 1 00:00 08123100010202.TLG
-rw-r--r-- 1 ins users 930495 Feb 10 10:00 sqlnet.log
-rw-r--r-- 1 ins users 195611078 Feb 11 00:00 09021000010202.TLG
-rw-r--r-- 1 ins users 183961342 Feb 12 00:00 09021100010202.TLG
-rw-r--r-- 1 ins users 195541787 Feb 13 00:00 09021200010202.TLG
-rw-r--r-- 1 ins users 188403819 Feb 14 00:00 09021300010202.TLG
-rw-r--r-- 1 ins users 180083539 Feb 15 00:00 09021400010202.TLG
-rw-r--r-- 1 ins users 174935554 Feb 16 00:00 09021500010202.TLG
-rw-r--r-- 1 ins users 184086392 Feb 17 00:00 09021600010202.TLG
-rw-r--r-- 1 ins users 183132340 Feb 18 00:00 09021700010202.TLG
-rw-r--r-- 1 ins users 204216062 Feb 19 00:00 09021800010202.TLG
-rw-r--r-- 1 ins users 197424799 Feb 20 00:00 09021900010202.TLG
-rw-r--r-- 1 ins users 194341059 Feb 21 00:00 09022000010202.TLG
-rw-r--r-- 1 ins users 184951088 Feb 22 00:00 09022100010202.TLG
-rw-r--r-- 1 ins users 163008982 Feb 23 00:00 09022200010202.TLG
-rw-r--r-- 1 ins users 187907245 Feb 24 00:00 09022300010202.TLG
-rw-r--r-- 1 ins users 210837242 Feb 25 00:00 09022400010202.TLG
-rw-r--r-- 1 ins users 186564560 Feb 26 00:00 09022500010202.TLG
-rw-r--r-- 1 ins users 214959454 Feb 27 00:00 09022600010202.TLG
-rw-r--r-- 1 ins users 209203339 Feb 28 00:00 09022700010202.TLG
-rw-r--r-- 1 ins users 201840157 Mar 1 00:00 09022800010202.TLG
-rw-r--r-- 1 ins users 173722179 Mar 2 00:00 09030100010202.TLG
-rw-r--r-- 1 ins users 221884036 Mar 3 00:00 09030200010202.TLG
-rw-r--r-- 1 ins users 210510974 Mar 4 00:00 09030300010202.TLG

thank u all for helping me out once again as i am not very good at NR and FNR which something very new to me.please bear with me and help me out.

Ok. Suppose you have only these two TLG files for December:

$ cat 08121500010202.TLG
203,0903100000042C,110,29,3,
207,0903100000042C,670,133,1
207,0903100000052C,36,25,7,0
203,0903100000052C,54,24,3,
203,0903100000052C,36,79,3,0
$
$ cat 08121400010202.TLG
203,0903100000042C,710,29,3,
207,0903100000042C,670,133,1
207,0903100000042C,26,9,7,09
207,0903100000052C,36,25,7,0
203,0903100000052C,720,24,3,
203,0903100000052C,36,79,3,0

Assume this is the subscriber list:

$ cat subscriber.list
1
2
3
110
218
26
36
42
50
54
670
710
720

Then this is the code & output:

nawk -F, 'NR==FNR{_[$0]=0;next}$1==203{a[$3]++}END{for(i in _){a==""&&a=0;print i" 203s = "a}}' subs.list 0812*.TLG
2 203s = 0
3 203s = 0
720 203s = 1
26 203s = 0
110 203s = 1
42 203s = 0
710 203s = 1
218 203s = 0
36 203s = 2
50 203s = 0
54 203s = 1
670 203s = 0
1 203s = 0

1. This shows for full month of December (0812*.TLG)
2. This shows count for subscriber which have 203 + those who are NOT in TLG file. No subscriber is missed.
3. It does the echo you wanted.

grep ^203 09022400010202.TLG | cut -d, -f3 | sort | uniq -c | while read line; do
cnt=$(echo $line | cut -d" " -f1)
acct=$(echo $line | cut -d" " -f2)
echo "Account $acct had $cnt 203's"
done

if you want to do each file then wrap it in a for loop

for file in $(ls `date +%y%m%d0010202.TLG`)
  echo "FILE: $file"
  grep ^203 $file | cut -d, -f3 | sort | uniq -c | while read line; do
    cnt=$(echo $line | cut -d" " -f1)
    acct=$(echo $line | cut -d" " -f2)
    echo "Account $acct had $cnt 203's"
  done
done

#!/bin/sh
set -x

for file in $(ls `date +%y%m%d0010202.TLG`)
echo "FILE: $file"
grep ^203 $file | cut -d, -f3 | sort | uniq -c | while read line; do
cnt=$(echo $line | cut -d" " -f1)
acct=$(echo $line | cut -d" " -f2)
echo "Account $acct had $cnt 203's"
done
done

when i run this script it gives
ins@ARTMGA01> ./arc.sh
./arc.sh: syntax error at line 4: `$' unexpected

nawk -F, 'NR==FNR{_[$0]=0;next}$1==203{a[$3]++}END{for(i in _){a[i]==""&&a[i]=0;print i" 203s = "a[i]}}' subs.list 0812*.TLG

This way we can only get the stats for combinely for whole month.

I think i am unable to make you understand what i need.i am sorry about that.

This is what i need
uary
i need the count of 203's for each day for each subcriberid for the whole month of feb

then i need to paste it in an excel file which i will do like these
subid 200 302 144 .....
date:
03-02-2009
04-02-2009
...........
.........2009
.........2009

thanks again for all the help

sorry on the first line add "; do"
for file in $(ls `date +%y%m%d0010202.TLG`); do

Based on your last email you will want to redirect the output into a text file and add the date to the output.
You can then import the text file into Excel as a {space} delimited file.

for file in $(ls 0902*0010202.TLG); do
  day=$(echo $file | cut -c 1-6)
  grep ^203 $file | cut -d, -f3 | sort | uniq -c | while read line; do
    cnt=$(echo $line | cut -d" " -f1)
    acct=$(echo $line | cut -d" " -f2)
    echo "Date 20${day} Account ${acct} had ${cnt} 203's" >> Feb_report.txt
  done
done

ins@ARTMGA01> ./arc.sh
./arc.sh: syntax error at line 4: `$' unexpected

its still giving me a syntax error

your error reports line 4 has a syntax error you need to provide more information. Paste your version of the script. Also some versions of shell do not like the use of "$()" so replace them with back ticks "`" usually found on the tilde "~" key.

ins@ARTMGA01> more arc.sh

#!/bin/sh

set -x

for file in $(ls 0902*0010202.TLG); do

day=$(echo $file | cut -c 1-6)

grep ^203 $file | cut -d, -f3 | sort | uniq -c | while read line; do

cnt=$(echo $line | cut -d" " -f1)

acct=$(echo $line | cut -d" " -f2)

echo "Date 20${day} Account ${acct} had ${cnt} 203's" >> Feb_report.txt

done

done