script to pull info from my email?

xinix · April 1, 2009, 7:06pm

Hi, I need help writing a script that would pull info from an email inbox and add it to an Evolution Calendar. I'm pretty sure I can google the commands to add the info to the calendar. The part that I really need help with is getting the info from the email into the command. Basically for work, when I accept an assignment, I get an email from an automated system that provides the job details. I'd like to pull the "who", "when" and "where" info and add that to the calendar. The emails are formated in a consistent fashion so the info I need will always be given in the same way.
Any help would be greatly appreciated as I'm new scripting and commands like 'grep'

Thank you.

xinix · April 1, 2009, 11:40pm

Ok so I'm thinking that AWK might be the best way to go but I have run into a problem that I'm sure has a solution. Basically I don't know how to use a string with spaces. I'll elaborate a bit more on what I'd like to do. Basically the body of the emails is similar to this:

You have been assigned a job starting on 01/23/4567.

The following are the details of the job:

*************
 Job Summary 
*************
Starting on              : 01/23/4567
Place                   : Name Of Place                   
Person                  : Lastname, Firstname


**********
 Job Days 
**********
Place                                             Date     From    To     
--------------------------------------------- -------- ------- -------
Name Of Place                                 01/23/45  8:20AM  3:00PM


Please do not reply to this system generated message.

I can get AWK to give me the name of place with:

awk '/Place :/ {print $3,$4,$5}' testfile

This gives me the output "Name Of Place" which is good, but the next thing I need to do is get the date and time. This info is on a line further down. I hoped that this command would work, but it doesn't:

awk '/Name Of Place/ {print $4,$5,$6}' testfile

it returns this output:

Of Place
01/23/45  8:20AM  3:00PM

What do I need to modify so that it only gives the date and time?

cfajohnson · April 2, 2009, 2:32am

awk '/^Name Of Place/ {print $(NF-2),$(NF-1),$NF}' testfile

xinix · April 2, 2009, 9:38am

I tried the suggested command and didn't get any output, but it got me on the right track. So, thank you very much.
It was the ^ that I was missing.

awk '/^Name Of Place/ {print $4,$5,$6}' testfile

This gives me the date and time info.

At least now I know how to get the data I need from a testfile. Next I need to figure out how to get it from my inbox. This shouldn't be too tricky as I'm guessing that there are quite a few posts in this forum that deal with email inboxes.

xinix · April 8, 2009, 12:14am

Hi, I've had some success at writing this script. I'm sure it's not the most graceful and it still needs some work, but, it's a good start. I've had to search quite a bit for things that I'm sure are fairly simple to do. I could use some help adding a bit of functionality that I need in the script.
So far I have the emails coming into a directory and a separate file is made for each one (getmail does this). This script goes through the directory and gets the info I need from the email and adds this to a calendar file(.ics). Right now I have it confined to a single test file until I can have it do everything it needs to do. Once I have it ready to use on all files that come, I'll add code for it to move the parsed email out and work through all the files in the directory until it's empty. The script checks a few line that are in the emails but aren't in the example I gave before.

Heres the script:

#!/bin/bash

NEWEST=$(ls -lrt | awk '{ f=$NF }; END{ print f }')
TMPNAME=$(awk '/Confirmation No.         :/ {print $4}' $NEWEST)
PLACE=$(awk '/Place                   :/ {print $3,$4,$5,$6,$7}' $NEWEST)
WHO=$(awk '/Person                  :/ {print $4,$5,$3}' $NEWEST)
TITLE=$(awk '/Title                    :/ {print $3,$4,$5,$6,$7}' $NEWEST)
DATE=$(awk "/^$PLACE/ "' {print $(NF-2)}' $NEWEST)
STIME=$(awk "/^$PLACE/ "' {print $(NF-1)}' $NEWEST)
ETIME=$(awk "/^$PLACE/ "' {print $(NF)}' $NEWEST)
DTSTART=$(date --date "$DATE $STIME +5 hours" +"%Y%m%d%H%M%S")
DTEND=$(date --date "$DATE $ETIME +5 hours" +"%Y%m%d%H%M%S")
DSTAMP=$(date --date '+5 hours' +"%Y%m%d%H%M%S")


echo -e "BEGIN:VEVENT\n\
DTSTART:$DTSTART\n\
DTEND:$DTEND\n\
DSTAMP:$DSTAMP\n\
UID:$TMPNAME-Sync-Script\n\
CLASS:PRIVATE\n\
CREATED:$DSTAMP\n\
DESCRIPTION:$TITLE\n\
LAST-MODIFIED:$DSTAMP\n\
LOCATION:$PLACE\n\
SEQUENCE:0\n\
STATUS:CONFIRMED\n\
SUMMARY:$WHO\n\
TRANSP:OPAQUE\n\
BEGIN:VALARM\n\
ACTION:DISPLAY\n\
DESCRIPTION:This is an event reminder\n\
TRIGGER:-P0DT0H30M0S\n\
END:VALARM\n\
END:VEVENT" > $TMPNAME

awk '{if (/END:VCALENDAR/) system("cat '$TMPNAME'"); print}' basic.ics > $TMPNAME.ics

mv $TMPNAME.ics basic.ics
rm $TMPNAME

This work well (mostly) and leaves me with a file that I can use with my calendar (I've decided against evolution). I say this works well because normally the email wil have only one line like this:

Name Of Place                                 01/23/45  8:20AM  3:00PM

But sometimes there will be more than one, e.g.:

Name Of Place                                 01/23/45  8:20AM  3:00PM
Name Of Place                                 01/24/56  8:20AM  3:00PM

The script gets the job date, start and end time from this line and it works well when there is only one line. Sometimes (not too often) the job will be on multiple consecutive days. So this line will be repeated with a different date.

How can I have the script add an entry for each line (day)?

cfajohnson · April 8, 2009, 8:35pm

xinix:

Hi, I've had some success at writing this script. I'm sure it's not the most graceful and it still needs some work, but, it's a good start. I've had to search quite a bit for things that I'm sure are fairly simple to do. I could use some help adding a bit of functionality that I need in the script.
So far I have the emails coming into a directory and a separate file is made for each one (getmail does this). This script goes through the directory and gets the info I need from the email and adds this to a calendar file(.ics). Right now I have it confined to a single test file until I can have it do everything it needs to do. Once I have it ready to use on all files that come, I'll add code for it to move the parsed email out and work through all the files in the directory until it's empty. The script checks a few line that are in the emails but aren't in the example I gave before.

Heres the script:
#!/bin/bash

NEWEST=$(ls -lrt | awk '{ f=$NF }; END{ print f }')

Why are you reversing the output of ls and then getting the last line instead of just getting the first?

NEWEST=$( ls -t | head -1 )

Or:

set -f; set -- $( ls -t )
NEWEST=$1

TMPNAME=$(awk '/Confirmation No.         :/ {print $4}' $NEWEST)
PLACE=$(awk '/Place                   :/ {print $3,$4,$5,$6,$7}' $NEWEST)
WHO=$(awk '/Person                  :/ {print $4,$5,$3}' $NEWEST)
TITLE=$(awk '/Title                    :/ {print $3,$4,$5,$6,$7}' $NEWEST)
DATE=$(awk "/^$PLACE/ "' {print $(NF-2)}' $NEWEST)
STIME=$(awk "/^$PLACE/ "' {print $(NF-1)}' $NEWEST)
ETIME=$(awk "/^$PLACE/ "' {print $(NF)}' $NEWEST)

Why seven calls to awk instead of one?

I don't have time to rewrite the awk code at the momment, but use this for a model:

eval "$( awk '{ printf "var" ++n "=\"%s\"\n", $1 }' "$NEWEST")"


DTSTART=$(date --date "$DATE $STIME +5 hours" +"%Y%m%d%H%M%S")
DTEND=$(date --date "$DATE $ETIME +5 hours" +"%Y%m%d%H%M%S")
DSTAMP=$(date --date '+5 hours' +"%Y%m%d%H%M%S")

echo -e "BEGIN:VEVENT\n\
DTSTART:$DTSTART\n\
DTEND:$DTEND\n\
DSTAMP:$DSTAMP\n\
...
TRIGGER:-P0DT0H30M0S\n\
END:VALARM\n\
END:VEVENT" > $TMPNAME

Why all the unnecessary \ns and backslashes?

echo "BEGIN:VEVENT
DTSTART:$DTSTART
DTEND:$DTEND
DSTAMP:$DSTAMP
...
TRIGGER:-P0DT0H30M0S
END:VALARM
END:VEVENT" > $TMPNAME

awk '{if (/END:VCALENDAR/) system("cat '$TMPNAME'"); print}' basic.ics > $TMPNAME.ics

mv $TMPNAME.ics basic.ics
rm $TMPNAME
This work well (mostly) and leaves me with a file that I can use with my calendar (I've decided against evolution). I say this works well because normally the email wil have only one line like this:
Name Of Place                                 01/23/45  8:20AM  3:00PM
But sometimes there will be more than one, e.g.:
Name Of Place                                 01/23/45  8:20AM  3:00PM
Name Of Place                                 01/24/56  8:20AM  3:00PM 
The script gets the job date, start and end time from this line and it works well when there is only one line. Sometimes (not too often) the job will be on multiple consecutive days. So this line will be repeated with a different date.

How can I have the script add an entry for each line (day)?

How do you add an entry to your calendar?

xinix · April 8, 2009, 11:19pm

Thanks for some of the comments.

Basically it was the first working code I found online. Your suggestion is better.

Each one gets a unique piece of info that is used for a specific line in the echo command. If there's a single call to that can do it that would be nice. But I just started learning awk stuff a week ago so I'm using what I can get to work for me (even if it might be a bit barbaric).

Basically:

PLACE=$(awk '/Place                   :/ {print $3,$4,$5,$6,$7}' $NEWEST)

This command gets me the name of the place. I need to know this first because some of the important information is on a line that starts with that name.

DATE=$(awk "/^$PLACE/ "' {print $(NF-2)}' $NEWEST)

This line gets the mm/dd/yy date from the source file and another call gets the start hour.
This needs to be converted to a yyyymmdd format and time +5 hours, which is done by this:

DTSTART=$(date --date "$DATE $STIME +5 hours" +"%Y%m%d%H%M%S")

This is repeated to get the finish time.
I'm sure I could consolidate some of these commads. Maybe once I'm a little more experienced.

The "\n\" are not needed.

The calendar uses a "filename.ics" file (basically an ical file) that several calendar apps seem to understand. This script adds the block of lines that make up an event entry just before the last line which closes the file.
the block looks something like this:

BEGIN:VEVENT
DTSTART:20090504170000
DTEND:20090504203000
DSTAMP:20090409021153
UID:34533016-Sync-Script
CLASS:PRIVATE
CREATED:20090409021153
DESCRIPTION:Descripion   
LAST-MODIFIED:20090409021153
LOCATION:Name of Place 
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:Name of Person
TRANSP:OPAQUE
BEGIN:VALARM
ACTION:DISPLAY
DESCRIPTION:This is an event reminder
TRIGGER:-P0DT0H30M0S
END:VALARM
END:VEVENT