To get older than last 7days records using awk scripting to generate report and send email

Hello All,

I have need as below:
1--> I need to get all users(who submit jobs) and their details by using below command:

qstat -u \*

output of the above command looks line below:

job-ID	prior	name	user-id	state	"submit/start at"	queue	jclass	slots ja-task-ID
-------------------------------------------------------------------------------------------
147956 10.29988 python_val user1       r     07/19/2017 07:30:15 queue1	1
147960 10.29988 python_dev user1       r     07/19/2017 07:30:33 queue2	1
147504 0.31988 gbdt_1.4.1 user2   r     07/19/2017 05:41:20 queue3 	1
122900 0.30988 run_python user3      r     07/10/2017 01:10:01 queue4 	1

The above output is space separated and i am interested in job-id, user-id and submit/start at (which is job submit date) from the above output

2--> I have another command which displays all users details from ADS as below:

/ccore/pbis/bin/enum-members "adsusers"

The output of the above command looks like below: (i am displaying one user's output, complete output contains bulk of below same blocks)

User object [001] (XXXXXXXX)
============
Enabled: 
Distinguished name: 
SAM account name: 
NetBIOS domain name: 
UPN: email@FINCO.COM
Display Name:FName LName
Alias: user-id
UNIX name:
GECOS:
Shell: 
Home directory: 
Windows home directory: 
Local windows home directory:
UID: 
Primary group SID: 
Primary GID: 
Password expired: 
Password never expires: 
Change password on next logon:
User can change password:
Account disabled:
Account expired:
Account locked:

In the above output i am interested in UPN (email), Display Name, Alias (user-id) from the above output

3 --> Now the requirement is i need to get the job details which are running more than last 7 days and send those details to user (which i will get email id from point#2)

currently i am getting this by following script:


!#/bin/bash

for line in `qstat -u \* | awk -v dt="$(date "--date=$(date) -7 day" +%m/%d/%Y)" 'BEGIN{OFS=":"}/qw/{if($6<dt) print $4,$1}'`;  
do 
user=$(echo $line | awk -F":" '{print $1}'); 
jobid=$(echo $line | awk -F":" '{print $2}');
email=$(/ccore/pbis/bin/enum-members "adsusers" | grep ^UNIX -B3  | grep $user -B2 | grep UPN | awk '{print $2}'); 
userName=$(/ccore/pbis/bin/enum-members "adsusers" | grep ^UNIX -B3  | grep $user -B2 | grep -i "Display Name" | awk -F":" '{print $2}')
echo -e "Hello $userName,\n\n Job(s) with job id(s): $jobid executing more than last 7 days, hence request you to take action, else job will be killed in another 1 days, \n\n Thank you." | mailx -s "Long running job for user: $userName ($user) and Job ID: $jobid" "${email}"
done

I know i am using a pattern matching using qw which i had to do and i am getting output as below:

Email Subject: Long running job for user:  FName LName (user-id) and Job ID(s): job-id
Email Body:
Hello FName LName,
Job(s) with job id(s): job-id executing more than last 7 days, hence request you to take action, else job will be killed in another 1 days, 
Thank you.

How can i implement the above script using awk? I know I am using awk, but its in bits and peaces.
Also the issue with the above script is - if the user submit more than one job, for example 2 in above case (user1 with job id-147956 and 147960), he is receiving 2 separate emails, but he should receive one email with 2 job details.

can some one please help me to fix this?

Thank you!!

Comparing dates in above format will fail on month or year transition, so you need a different approach. One would be converting to epoch seconds and then compare, or, as done here, just shifting the number to year/month/day format.
Unfortunately, none of the user-ids given in the qstat output find a match in the ADS file, so assumptions had to be made.
Try the following - ON A TEST SYSTEM, NOT PRODUCTION! - (but be aware that there's an small uncertainty in mailing the body as this can't be tested from here):

{ qstat -u \*; /ccore/pbis/bin/enum-members "adsusers"; } | awk -v dt="$(date "--date= -7 day" +%Y%m%d)" '

/^User obj/     {F2 = 1
                 FS = ":"
                 T1 = T2 = ""
                 next
                }
!F2             {if (NR < 3) next
                 split ($6, T, "/")
                 if (T[3]T[1]T[2] < dt) JID[$4] = JID[$4] " " $1
                 next
                }

/^UPN/          {T1 = $2
                }
/^Display/      {T2 = $2
                }
/^Alias/        {gsub (/ /, _, $2)
                 EM[$2] = T1
                 DN[$2] = T2
                }
END             {for (j in JID) {print "echo Hello " DN[j] "\\" RS "\\"
                                 print "Job\(s\) with job id\(s\): " JID[j] " executing more than last 7 days, hence request you to take action, else job will be killed in another
                                 print "Thank you.\\"
                                 print " \"|\" "
                                 print "mailx -s \"Long running job for user: " DN[j] " (" j ") and Job ID: " JID[j] "\" "  EM[j] 
                                }
                }
' | sh
1 Like

Hello RudiC, Thank you so much for your time and response.
All the users of

 qstat -u \* 

will definitely present in

 /ccore/pbis/bin/enum-members "adsusers" 

for sure.
Which means what ever user-id's as part of 1st command output will definitely be present (all the details including user-id (Alias)) in 2nd command output.
I have slightly change your script to test by adding another awk variable - testEmail like below:

{ qstat -u \*; /ccore/pbis/bin/enum-members "adsusers" } | awk -v dt="$(date "--date= -7 day" +%m/%d/%Y)" -v testEmail="vasubabu1@finco.com"'

/^User obj/     {F2 = 1
                 FS = ":"
                 T1 = T2 = ""
                 next
                }
!F2             {if (NR < 3) next
                 split ($6, T, "/")
                 if (T[3]T[1]T[2] < dt) JID[$4] = JID[$4] " " $1
                 next
                }

/^UPN/          {T1 = $2
                }
/^Display/      {T2 = $2
                }
/^Alias/        {gsub (/ /, _, $2)
                 EM[$2] = T1
                 DN[$2] = T2
                }
END             {for (j in JID) {print "echo Hello " DN[j] "\\" RS "\\"
                                 print "Job\(s\) with job id\(s\): " JID[j] " executing more than last 7 days, hence request you to take action, else job will be killed in another
                                 print "Thank you.\\"
                                 print " \"|\" "
                                 print "mailx -s \"Long running job for user: " DN[j] " (" j ") and Job ID: " JID[j] "\" "  testEmail 
                                }
                }
' | sh

But i am getting below exception:

awk: warning: escape sequence `\(' treated as plain `('
awk: warning: escape sequence `\)' treated as plain `)'
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
POSIX options:          GNU long options:
        -f progfile             --file=progfile
        -F fs                   --field-separator=fs
        -v var=val              --assign=var=val
        -m[fr] val
        -O                      --optimize
        -W compat               --compat
        -W copyleft             --copyleft
        -W copyright            --copyright
        -W dump-variables[=file]        --dump-variables[=file]
        -W exec=file            --exec=file
        -W gen-po               --gen-po
        -W help                 --help
        -W lint[=fatal]         --lint[=fatal]
        -W lint-old             --lint-old
        -W non-decimal-data     --non-decimal-data
        -W profile[=file]       --profile[=file]
        -W posix                --posix
        -W re-interval          --re-interval
        -W source=program-text  --source=program-text
        -W traditional          --traditional
        -W usage                --usage
        -W use-lc-numeric       --use-lc-numeric
        -W version              --version

To report bugs, see node `Bugs' in `gawk.info', which is
section `Reporting Problems and Bugs' in the printed version.

gawk is a pattern scanning and processing language.
By default it reads standard input and writes standard output.

Examples:
        gawk '{ sum += $1 }; END { print sum }' file
        gawk -F: '{ print $1 }' /etc/passwd

Could you please help me to fix this? Thank you!!

---------- Post updated at 11:12 AM ---------- Previous update was at 10:28 AM ----------

I got what the problem is, in below print statement double quotes was not ending properly.

print "Job\(s\) with job id\(s\): " JID[j] " executing more than last 7 days, hence request you to take action, else job will be killed in another

Unfortunately even after i fix this i didn't receive any email,and I don't see any exceptions. Just executing and I don't see any email, that's all. HEre is the last few statements of output:

> END             {for (j in JID) {print "echo Hello " DN[j] "\\" RS "\\"
>                                  print "Job(s) with job id(s): " JID[j] " executing more than last 7 days, hence request you to take action, else job will be killed in another 1 Day"
>                                  print "Thank you.\\"
>                                  print " \"|\" "
>                                  print "mailx -s \"Long running job for user: " DN[j] " (" j ") and Job ID: " JID[j] "\" "  testEmail
>                                 }
>                 }
> ' | sh
[vasulr@assbd0035 ~]$

could you please help me? Thank you!!

I'm afraid I can't. It was quite a mess to get the output in shape, and I couldn't test it with mail.
Try not to pipe it into sh , but analyse it on screen if the output makes sense. If it does, create a file of the same contents and have sh run it, mayhap with -xv options set. You may want to come back with results / errors so thes can be discussed here.

EDIT: looks like a partial line 1 days," was lost in dragging and dropping the script. Sorry for that - I counted double quotes incredibly often before it (almost) worked.

Hello RudiC,

I have made small changes in the solution that you provided and now, its working as expected, Thank you so much for your time.

Here is the final script that is working as expected:

!#/bin/bash

{ qstat -u \*; /ccore/pbis/bin/enum-members "adsusers"; } | awk -v dt=$(date "--date=$(date) -7 day" +%m/%d/%Y) '
 /^User obj/     {
                  F2 = 1
                  FS = ":"
                  T1 = T2 = ""
                  next
                 }
 !F2             {
                  if (NR < 3) next
                  if ($5 ~ "qw" && $6 < dt) JID[$4] = $1 "," JID[$4]
                  next
                 }

 /^UPN/          {T1 = $2
                 }
 /^Display/      {T2 = $2
                 }
 /^Alias/        {gsub (/ /, _, $2)
                  EM[$2] = T1
                  DN[$2] = T2
                 }
 END             {for (j in JID) {print "echo -e \"Hello " DN[j] " \\n \\nJob(s) with job id(s): " JID[j] " executing more than last 7 days, hence request you to take action, else job(s) will be killed in another 1 day \\n \\n Thank you.\" | mailx -s \"Long running job for user: " DN[j] " (" j ") and Job ID(s): " JID[j] "\" " EM[j]
                                 }
                 }
 ' | sh

Thank you!!
Vasu

You know that it won't work reliably as given in your post#5? With that date format, January 2018 is less (earlier) than December 2017.
And, you don't need the $(date) for the date arithmetics. date assumes NOW when differences are given.
The trick with the one line printout of the commands for sh is a good one...