Help with Email Status shell script

unimaxlin · January 19, 2011, 4:49pm

I have written a bash script to to sort the data from logs i need some help in printing the outputs , i dont have much ideas in bah scripting.

Sample script
-----------------------

#!/bin/bash
a=`date | cut -d " " -f2,2,3`
cat /var/log/maillog |grep "$a" |grep -E -e 'deferred|bounced' >> /tmp/mailer
for i in `cat /tmp/mailer`
do
address=`echo $i|awk -F " " '{print $7}' /tmp/mailer`
result=`echo $i|awk -F " " '{print $12}' /tmp/mailer`
reason=`echo $1|awk -F " " '{print $17,$18,$19,$20,$21,$22,$23,$24,$25,$26,$28,$29}' /tmp/mailer`
echo "$address $result $reason"
done

Present output
----------------------------
the email send to=<mhillers@bellsouth.net>, status=deferred talk to me: 521-75.126.169.21 blocked by ldap:ou=rblmx,dc=att,dc=net 521 Error - for abuse.
the email send to=<timber'lee@timberlee.com>,status=deferred timeout (#4.4.2) (in reply to end of DATA command))
the email sendto=<apache@bikepartsusa.com>,status=bounced The email account that you tried to reach does not Please try

I want and output Like this any Ideas

mhillers@bellsouth.net status=deferred Error - for abuse
timber'lee@timberlee.com status=deferred timeout
apache@bikepartsusa.com email account that you tried to reach does not

---------- Post updated at 04:49 PM ---------- Previous update was at 04:44 PM ----------

I am sending the log format i have
/var/log/maillog

Jan 19 12:21:10 bikepartsusa postfix/smtp[6054]: 4DD75D4C45: to=<rfgoodman@bellsouth.net>, relay=gateway-f2.isp.att.net[207.115.11.16]:25, delay=155893, delays=155892/0.01/0.2/0, dsn=4.0.0, status=deferred (host gateway-f2.isp.att.net[207.115.11.16] refused to talk to me: 521-75.126.169.21 blocked by ldap:ou=rblmx,dc=att,dc=net 521 Error - Blocked for abuse. Contact abuse_rbl@abuse-att.net.)

Jan 19 12:29:30 bikepartsusa postfix/smtp[6756]: 57986D4C53: to=<apache@bikepartsusa.com>, relay=ASPMX.L.GOOGLE.com[74.125.159.27]:25, delay=3.5, delays=0.01/0/0.3/3.1, dsn=5.1.1, status=bounced (host ASPMX.L.GOOGLE.com[74.125.159.27] said: 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the recipient's email address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1 'This Gmail user does not exist...' - Gmail Help g9si15796109anh.79 (in reply to RCPT TO command))

Jan 19 12:21:31 bikepartsusa postfix/smtp[6055]: 9E874D4C43: to=<timber'lee@timberlee.com>, relay=nullmx.timberlee.com[209.181.247.105]:25, delay=423999, delays=423979/0.01/0.08/20, dsn=4.0.0, status=deferred (host nullmx.timberlee.com[209.181.247.105] said: 451 timeout (#4.4.2) (in reply to end of DATA command))

Chubler_XL · January 19, 2011, 7:58pm

OK you should probably start by extracting the 3 data fields you need from the log (toemail, status and errorstr):

awk '
   /to=/&&/, status=.* \(/ {
   toemail=gensub(".* to=<(.*)>.*", "\\1", $0);
   status=gensub(".*, status=([^ ]*) .*", "\\1", $0);
   errorstr=gensub(".*, status=[^ ]* \\((.*)\\)$", "\\1", $0);
   print toemail, status, errorstr
} ' /var/log/maillog

It then looks like you want to suppress display of the bounced status and transpose/convert the errorstr.

The errorstr seems to be quite free format and I suspect there are a many more types of errorstr than you example data lists so I've left the string as it appears in the original logfile. Perhaps you can give us some more indication on what should happen with this string.

unimaxlin · January 20, 2011, 3:26pm

i am getting an error when i execute this directly can

[root@freebsd:/root 01:42]sh test2.sh
awk: calling undefined function gensub
input record number 1, file /var/log/maillog
source line number 3

---------- Post updated at 03:26 PM ---------- Previous update was at 03:17 PM ----------

My present out put is like this i need it to setup the script in a single line

to=<mhillers@bellsouth.net>,
to=<skimmels@bellsouth.net>,
to=<rfgoodman@bellsouth.net>,
status=deferred
status=deferred
status=deferred
timeout (#4.4.2) (in reply to end of DATA command))
The email account that you tried to reach does not Please try
#5.1.0 Address rejected www@www.unitedlabsupplies.com (in reply to RCPT TO command))

output should be like this

mhillers@bellsouth.net status=deferred Error - for abuse

please help me i creating my first script

Chubler_XL · January 20, 2011, 4:56pm

OK you don't have gnu awk this should be equilivent:

awk '
   /to=/&&/, status=.* \(/ {
   toemail=$0;
   status=$0;
   errorstr=$0;
   gsub(".* to=<", "", toemail);
   gsub(">.*", "", toemail);
   gsub(".*, status=", "", status);
   gsub(" .*", "", status);
   gsub(".*, status=[^ ]* \\(", "", errorstr);
   gsub("\\)$", "", errorstr);
   print toemail, status, errorstr
} ' /var/log/maillog

unimaxlin · January 20, 2011, 7:27pm

the present output will not help me it similar to log i need only email id status the a small reason
present output
-------------------
mhillers@bellsouth.net deferred host gateway-f2.isp.att.net[207.115.11.16] refused to talk to me: 521-75.126.169.21 blocked by ldap:ou=rblmx,dc=att,dc=net 521 Error - Blocked for abuse. Contact abuse_rbl@abuse-att.net.
www@www.unitedlabsupplies.com bounced host smtp.secureserver.net[216.69.186.201] said: 550 #5.1.0 Address rejected www@www.unitedlabsupplies.com (in reply to RCPT TO command)
apache@bikepartsusa.com bounced host ASPMX.L.GOOGLE.com[74.125.159.27] said: 550-5.1.1 The email account that you tried to reach does not exist. Please try 550-5.1.1 double-checking the recipient's email address for typos or 550-5.1.1 unnecessary spaces. Learn more at 550 5.1.1 ogle.com/support/bin/answer.py?answer=6596]'This Gmail user does not exist...' - Gmail Help g9si15796109anh.79 (in reply to RCPT TO command)

i need an output like this

mhillers@bellsouth.net status=deferred Error - for abuse
timber'lee@timberlee.com status=deferred timeout
apache@bikepartsusa.com email account that you tried to reach does not

Chubler_XL · January 20, 2011, 7:47pm

awk '
   /to=/&&/, status=.* \(/ {
   status="status=deferred";
   toemail=$0;
   errorstr=$0;
   gsub(".* to=<", "", toemail);
   gsub(">.*", "", toemail);
   gsub(".*, status=[^ ]* \\(", "", errorstr);
   gsub("\\)$", "", errorstr);
   if(index(errorstr, "abuse")) errorstr="Error - for abuse";
   if(index(errorstr, "not exist")) { 
      status="email" ; 
      errorstr="account that you tried to reach does not"
   }
   if(index(errorstr, "timeout")) errorstr="timeout";
   print toemail, status, errorstr
} ' /var/log/maillog

unimaxlin · January 21, 2011, 2:25pm

Thank`s for great help i need to parse a few more log files

present output
----------------------
drlarocque@juno.com status=deferred host mx.dca.untd.com[64.136.44.37] refused to talk to me: 550 Access denied...181d58d961d8bdbd296888915959596869c55d0878294c2dbd6d6c5d25ac5d0d55a1d5550decec78b1ecf9852d192165196d01080c016d018d4c...
scofieldserco@juno.com status=deferred delivery temporarily suspended: host mx.dca.untd.com[64.136.44.37] refused to talk to me: 550 Access denied...4f5135b57185a9a96160c1a1bd80bda1b9053d592061758da9b0643dcd103d95e984e5e995909020e190241d8d00f1f000b0f959c9f9b0f95075...
pgstore@post-gazette.com status=deferred host mailrelay3.post-gazette.com[64.206.167.180] said: 552 Rejected by CA Gateway Security [scm-3.post-gazette.com] (ASF-99) (in reply to end of DATA command)
pgstore@post-gazette.com status=deferred host mailrelay3.post-gazette.com[64.206.167.180] said: 552 Rejected by CA Gateway Security [scm-3.post-gazette.com] (ASF-99) (in reply to end of DATA command)
biostockroom@umbc.edu status=deferred host mxin.umbc.edu[130.85.12.6] said: 451 4.3.2 Greylisting. Try back in a while. (in reply to RCPT TO command)

i would like to parse it
---------------------------------
i wan to parse a few more logs

biostockroom@umbc.edu status=deferred Greylisting
pgstore@post-gazette.com status=deferred host Rejected by CA Gateway Security
scofieldserco@juno.com status=deferred delivery temporarily suspended
drlarocque@juno.com status=deferred host refused to talk to me:

Chubler_XL · January 23, 2011, 5:05pm

I suspect you are going to find a lot more of these so probably best to have a replace_list file.

Create a replace_list file for yourself with 3 ; separated fields

Search string
New status
New error string

If 1. is found in your error string the string is replaced with 3. also if 2. is non-blank then status is replaced with 2.

For you requirements so far you need this replace_list:

abuse;;Error - for abuse
not exist;email;account that you tried to reach does not
timeout;;timeout
Greylisting;;Greylisting
delivery temporarily suspended;;delivery temporarily suspended
refused to talk;;host refused to talk to me:
CA Gateway Security;;Rejected by CA Gateway Security

New script to load replace list and output your required strings:

awk '
   NR == FNR { R[i++]=$0; next }
   /to=/&&/, status=.* \(/ {
   status="status=deferred";
   toemail=$0;
   errorstr=$0;
   gsub(".* to=<", "", toemail);
   gsub(">.*", "", toemail);
   gsub(".*, status=[^ ]* \\(", "", errorstr);
   gsub("\\)$", "", errorstr);
   for (i in R) {
       split(R, F, ";");
           if(index(errorstr, F[1])) {
               errorstr=F[3];
               if (F[2]!="") status=F[2];
           }
   }
   print toemail, status, errorstr
} ' replace_list mail.log

unimaxlin · January 24, 2011, 8:34am

Thanks for excellent help Chubler_XL

i am getting an error while i run this script

[dispatc 05:04 AM| ~]# sh test.sh
wk: not found

i have created a new file replace_list
touch replace_list
vi replace_list
abuse;;Error - for abuse
not exist;email;account that you tried to reach does not
timeout;;timeout
Greylisting;;Greylisting
delivery temporarily suspended;;delivery temporarily suspended
refused to talk;;host refused to talk to me:
CA Gateway Security;;Rejected by CA Gateway Security

paste these lines and after that i modify the log location to /var/log/maillog from mail.log then i am getting this error i hav no idea that what is wrong here

---------- Post updated at 05:20 AM ---------- Previous update was at 05:08 AM ----------

Thanks a lot the script was working it was my mistake copy and paste i used wk insted of awk i am sorry and thanks for your great help

---------- Post updated at 05:23 AM ---------- Previous update was at 05:20 AM ----------

Hello Chubler_XL

I would like to exclude the line similer to this root@ how can do this please help me

root@localhost status=deferred delivered to file: /dev/null

---------- Post updated at 08:34 AM ---------- Previous update was at 05:23 AM ----------

please help me i would like to exclude the line with mail to root

Chubler_XL · January 24, 2011, 11:06pm

New line to exclude root@ emails added below (in blue)

awk '
   NR == FNR { R[i++]=$0; next }
   /to=<root@localhost/ { next }
   /to=/&&/, status=.* \(/ {
   toemail=$0;
   status=$0;
   errorstr=$0;
   gsub(".* to=<", "", toemail);
   gsub(">.*", "", toemail);
   status="status=deferred";
   gsub(".*, status=[^ ]* \\(", "", errorstr);
   gsub("\\)$", "", errorstr);
   for (i in R) {
       split(R, F, ";");
           if(index(errorstr, F[1])) {
               errorstr=F[3];
               if (F[2]!="") status=F[2];
           }
   }
   print toemail, status, errorstr
} ' replace_list /var/log/maillog