Behavior I am looking for:
$ for var in a 'b c'; do echo $var; done
a
b c
What I am getting when I try to use a variable:
$ test="a 'b c'"
$ for var in $test; do echo $var; done
a
'b
c'
Is there a workaround?
Mike
How about this:
$ test=(a 'b c')
$ for var in "${test[@]}"; do echo $var; done
a
b c
or
$ set a 'b c'
$ while [ $# -gt 0 ]; do echo $1; shift ; done
a
b c
This does not get around my problem of having variable input.
test=($some_some_variable_with_quotes)
breaks it.
Mike
---------- Post updated at 02:14 PM ---------- Previous update was at 02:02 PM ----------
I'm currently working on a workaround that uses \n
Whatever code is parsing arguments in bash works the way I want this to work
script Argument1 "Argument 2" 'Argument 3'
sets $1 $2 and $3 just as you would expect/want
I am trying to parse commands from STDIN or a file the same way.
Mike
I guess here you are trying to avoid using eval:
eval test=($some_some_variable_with_quotes)
Shell doesn't do that kind of doublethink. If something's in a variable, it's taken literally (with the exception of glob characters).
Just like in your other thread:
If you explain your actual goal -- not the means you've picked to accomplish it -- we could help you far better!
Your questions suggest you've taken a wrong turn and ended up building a shell-inside-the-shell kind of thing, instead of just using the shell's features in the first place (i.e. building a configuration-file-reader in the shell, instead of just sourcing the file). Not knowing what the shell can do often leads to this kind of catch-22 situation.
But for the record, the xargs utility can understand quotes and print properly grouped lines:
$ xargs -n 1 <<EOF
a b c d e
"1 2 3 4 5"
'q asdf ads f'
EOF
a
b
c
d
e
1 2 3 4 5
q asdf ads f
$
2 Likes
I thought I did just that above. I'll try again:
I have a file with argments in it. There can be multiple arguments per line. Arguments can contain whitespace (in which case they are quoted). I want to itterate through them and handle them (such as with a case statement). I want the itteration to separate on unqoted white space but not quoted whitespace.
Mike
PS. Will check out the evel suggestion.
PPS. Will check out the xargs suggestion too.
Never use eval if you can possibly avoid it. If your file contains `rm -Rf ~/` and you feed that text into eval, eval will execute that code!
You explained one tiny corner of one tiny corner of your problem. What exactly is this line-by-line loader thing for? Is it a part of anything larger?
xargs with a here doc seems to work quite nicley:
# test="a 'b c'"
# xargs -n 1 << EOF | while read var; do echo $var ; done
$test
EOF
a
b c
I am trying to make a stand-along mailer along the lines of the minimal one I put together Monday but with everything configurable and a lot more features. The body of the message comes from STDIN but I also want to be able to pass arguments to manipulate headers at the top of STDIN since the arguments line could get very crowded very fast!
The darned thing works. In fact it works great. I am just tying up loose ends. (multiple attachments, implementing getopts, etc.) Right now commands on STDIN need to be one per line and I want to get around that although I am beginning to think the lack of need for quoting of extended e-mail addresses might be a feature: "Public, John Q" JQPublic@domain.com
Draft of Man:
tmail - lean send-only mailer for scripts and minimal embedded systems
tmail [OPTION] [COMMANDs]
also accepts COMMANDs form STDIN if first line is "%%%%%".
Use "%%%%%" to signify the end of OPTION section (remainder is mail body).
Normally the end of the mail body is EOF but "%%%%%" can be used as well.
This is useful when calling tmail from the terminal.
Execution order is [OPTIONs] [COMMANDs] [COMMANDs from STDIN].
Any COMMAND without the ":" character "naked email" is assumed to be a email recipient.
See -b below.
Options without arguments:
-? Help Show usage and exit
-b Blind Mode Assume naked e-mails are BCC: instead of To:
-l Loud show the telnet session.
-x Debug Mode echo what would have gone to telnet to stdout
(also sets delay=0)
Options with arguments:
-d 'date/time' Date/Time override
Replaces calculated Date/Time with string.
(same as command date:'date/time')
-h 'host' Host override
Replaces how the computer introduces itself to the SMTP server.
(same as command host:'host')
-m 'boundary' MIME Boundary override
Replaces default MIME boundary
(same as command boundary:'boundary')
-v server:port:delay Server Override
Replaces the default SMTP Sever and Port number.
Delay is the delay after issuing commands to SMTP server.
See 'man sleep' for units.
Any field left blank does not override default.
':18:' or '::2" changes only the Port or Delay, respectively.
(same as command server:)
-a path:type:name Attach File
Add an attachment to the email. If not included name will be
the short name of the path and type will be guessed from the
file extension, if any.
(same as command attach: or a:)
-f filepath Use file for body of message instead of stdin.
-h 'header' Replaces "%%%%%" header with another string.
-s 'subject' Sets the message subject.
Commands:
from:email@domain Set Sender (also f:)
to:email@domain Add a "To:" recipient (also t:)
cc:email@domain Add a "CC:" recipient (also c:)
bcc:email@domain Add a recipient (also b:)
subject:'subject' Sets the message subject (see -s above) (also s:)
attach:path:type:name Add an attachment (see -a above) (also a:)
server:server:port:delay Set Server, Port, and Delay (see -v above)
port:port Set Port (does not change server or dealy)
delay:delay Set Delay (does not change server or port)
host:host Set Host (see -h)
boundary:'boundary' Set MIME boundary (see -m)
other Naked Email. Any command without a ":" is assumed to be a "naked email".
Treated as a "To:" recipient unless blind mode (-b) is set.
#! /bin/bash
##################################################################################
# tmail: Send an Email with Telnet #
##################################################################################
#
##################################################################################
# Declarations #
##################################################################################
header="%%%%%"
boundary="XOXOXOXOXOXOXOXOXOXOXOXOXOXOXOXOXOXOXOXOXO"
warning="This is a multipart message in MIME format."
mailDate="$(date +%a", "%d" "%b" "%Y" "%T" "%z)"
host="$HOSTNAME"
sender="$USER""@""$HOSTNAME"
##################################################################################
#Standard User Editable Defaults
#move this to a .conf file later
subject="no subject"
server="smtp.abcde.com"; port="25"; delay="1"
##################################################################################
declare -A recipients # We are going to use associative array keys (and not values)
declare -A toList # to keep track of mailing lists ${!array[@]} as a builtin
declare -A ccList # way of preventing duplicates without sorting.
# add getopts options to handle "-x" type arguments.
# grab remaining arguments
#commands+=" ""$@"" "
# read in commands from stdin between "$header" lines, if any
read -r line
if [[ "$line" == "$header" ]]; then
firstLine=""
while read line && [[ "$line" != "$header" ]]; do commands+=$"$line"$'\n'; done
else
firstLine="$line"
fi
# Process commands from whatever source
# echo "$commands" #debug
oIFS="$IFS"; IFS=$'\n'; for var in $commands; do # For does not seem to like temporary assignment
if [[ "$var" == *:* ]]; then
IFS=":" read -r comm val1 val2 val3 < <(echo "$var")
case "${comm,,}" in
"subject"|"s") subject="$val1" ;;
"from"|"f") sender="$val1" ;;
"to"|"t") recipients["$val1"]=1; toList["$val1"]=1 ;;
"cc"|"c") recipients["$val1"]=1; ccList["$val1"]=1 ;;
"bcc"|"b") recipients["$val1"]=1 ;;
"attach"|"a") # add this
;;
"server")
[[ "$val1" ]] && server="$val1"
[[ "$val2" ]] && port="$val2"
[[ "$val3" ]] && delay="$val3"
;;
"port") port="$val1" ;;
"delay") delay="$val1" ;;
"host") host="$val1" ;;
"boundary") boundary="$val1" ;;
"warning") warning="$val1" ;;
"date") mailDate="$val1" ;;
*) echo "bad command"; echo "var: ""$var"; echo "1: ""$val1"; echo "2: ""$val2"; echo "3: ""$val3";echo "4: ""$val4"; exit 1 ;; #add usage and exist here.
esac
else recipients["$var"]=1; [[ ! "$blind" ]] && toList["$var"]=1
fi
done; IFS="$oIFS"
# Telnet Session
{ echo "HELO ""$host"
sleep "$delay"
echo "MAIL FROM: ""$sender"
sleep "$delay"
oIFS="$IFS"; IFS=$'\n'; for i in "${!recipients[@]}"; do
echo "RCPT TO: ""$i"
sleep "$delay"
done; IFS="$oIFS"
echo "DATA"
sleep "$delay"
echo "From: ""$sender"
toString="To: "
oIFS="$IFS"; IFS=$'\n'; for i in "${!toList[@]}"; do
toString+="$i"", "; done; IFS="$oIFS"
echo "${toString%", "}"
ccString="CC: "
oIFS="$IFS"; IFS=$'\n'; for i in "${!ccList[@]}"; do ccString+="$i"", "; done; IFS="$oIFS"
echo "${ccString%", "}"
echo "Date: ""$mailDate"
echo "Subject: ""$subject"
echo "Content-Type: multipart/mixed; boundary=\"""$boundary""\""
echo "MIME-Version: 1.0"
echo
echo "$warning"
echo
echo "--"""$boundary""
echo "Content-Type: text/plain"
echo
if [[ "$firstLine" ]]; then echo "$firstLine"; fi
while read line && [[ "$line" != "$header" ]]; do echo "$line"; done
echo
#add a loop for multiple attachments
if [[ "$attachment" ]]; then # add an attachment if not blank
echo "--"""$boundary""
echo "Content-Type: ""$attachmentType""; name=\""$attachmentName"\""
echo "Content-Transfer-Encoding: base64";
echo "Content-Disposition: attachment; filename=\""$attachmentName"\""
echo
base64 <"$attachment" | head
echo
fi
echo "--"""$boundary"""--"
echo "."
sleep "$delay"
echo "QUIT"; } | telnet "$server" "$port"
exit 0
STDIN (or file when I implement that):
At this point I am taking advantage of the lack of a need to quote spaces by using the \n workaround.
%%%%%
subject:This is the mail subject
from:mike
to:bob
cc:frank
bcc:diane
%%%%%
Hello guys.
Here is an email
Mike
Mike
1 Like
I'd suggest using boundary=$(uuidgen -t) rather than that XOXO.. string as this will avoid issues when a email from this program is attached in another.
good idea. Did not think of email inside an email.
Not in the embeded environment I am designing for but I will grab a certain number of bytes from urandom and pass them into base64.
update: base64 ended up being very unpredictable length even with huge amount of binary date fed to it could be strangely short. I used 512 bytes into sha256sum for a reasonable length header.
Mike
It's not whether it "likes" it or not. It's because of the order shell processes a statement.
The temporary IFS assignment only becomes valid when the statement is executed -- but in order to execute the statement, it already did splitting on $VAR. read on the other hand doesn't need a pre-splitted variable.
1 Like
What you describe is a (very simple) parser. The following sketch roughly does what you want, you will want to put it into a function in you script. I wrote it for ksh initially, so you will probably have to adapt it somewhat ("print" -> "echo", etc.).
function f_Parse
{
typeset fIn="$1" # input file
typeset lInStr=0 # flag: inside (quoted) string
typeset chProfStr=""
typeset chAct="" # current char
typeset chRest=""
typeset chBuffer=""
chProfStr="$(cat "$fIn")"
while [ -n "$chProfStr" ] ; do
chRest="${chProfStr#?}" # split off first char
chAct="${chProfStr%${chRest}}"
chProfStr="$chRest"
#print - "chAct: $chAct"
#print - "read : $chBuffer"
case "$chAct" in
\")
if [ $lInStr -eq 0 ] ; then
(( lInStr = 1 ))
else
(( lInStr = 0 ))
fi
chBuffer="${chBuffer}${chAct}"
;;
" ")
if [ $lInStr -eq 0 ] ; then
print - "$chBuffer" # write / flush buffer
chBuffer=""
else
chBuffer="${chBuffer}${chAct}"
fi
;;
*)
chBuffer="${chBuffer}${chAct}"
;;
esac
done
return 0
}
I hope this helps.
bakunin
2 Likes
Love the f_Parse function Bakunin. It got me thinking about a quoted_read function that works like read but supports quoted strings so I could call it like this
test='one two three four five
"this and" that
done'
while quoted_read p1 p2 rest
do
printf "%s\n" "p1=$p1" "p2=$p2" "rest=$rest" ""
done <<EOF
$test
EOF
and get the output:
p1=one
p2=two
rest=three four five
p1=this and
p2=that
rest=
p1=done
p2=
rest=
And here is the function I came up with (based on f_Parse):
function quoted_read
{
typeset lInStr=0
typeset chBuffer
typeset chAct
typeset RET=1
[ $# -eq 0 ] && set -- REPLY
OIFS=$IFS
IFS=
while read -N 1 chAct
do
RET=0
case "$chAct" in
\") (( lInStr = (lInStr + 1) % 2 ))
continue
;;
" "|$'\n') if [ $lInStr -eq 0 ]
then
[ "$chAct" == $'\n' ] && break
if [ $# -gt 1 ]
then
# More vars to assign -> not appending to current
export $1="$chBuffer"
shift
chBuffer=""
continue
fi
fi
;;
esac
chBuffer="${chBuffer}${chAct}"
done
if [ ${#chBuffer} -gt 0 ]
then
export $1="$chBuffer"
shift
fi
# Blank any vars not assigned
while [ $# -gt 0 ]
do
export $1=""
shift
done
IFS=$OIFS
return $RET
}
1 Like
Guys I love the parser! It is not 100% clear to me how it handles multiple spaces in a row. As far as multiple carraige returns, the behavior of read is to include blank lines as output and the behavior or for with /n as the delimiter is to skip them. Both however, treat multiple unquoted spaces or tabs as if the were one.
I assume we can add another inString variable for single quotes (') and add $'\t' (tab) to the second case condition in Chubler_XL's version.
Hmm . . . I had kind of assumed that ' was "superior" to ". Actually whichever one comes first is "superior"
$ echo "'" "''" "'''"
' '' '''
$ echo '"' '""' '"""'
" "" """
$ echo '"'"'" "'"'"'
"' '"
So inStringSingle if active would have not not count " and inStringDouble if active would not count '.
Meanwhile, I have the mailer to the point that I can use it for the purpose I wrote it. I will return to it in the future to make it much nicer. I've decided that my \n workaround when I loop through the command list is actually a feature since so many e-mail strings and other things SMTP have spaces and will pass them one per line with spaces on STDIN. When I add options and $@, I will just add /n characters to the command string between $1, $2, etc. as BASH already parses quotes in arguments neatly.
Anyway, here is the script in working (but not complete form).
#! /bin/bash
##################################################################################
# tmail: Send an Email with Telnet v0.5 #
##################################################################################
# Michael E. Stora, July 2014
##################################################################################
# Declarations #
##################################################################################
header="%%%%%"
boundary="$(read -r -n 512 line </dev/urandom; echo $line | sha256sum)"
warning="This is a multipart message in MIME format."
host="$HOSTNAME"
sender="$USER""@""$HOSTNAME"
mailDate="$(date +%a", "%d" "%b" "%Y" "%T" "%z)"
##################################################################################
# Standard User Editable Defaults
# move this to a .conf file later <<<<<<<<<<<< To Do
subject="no subject"
server="smtp.xxxxx.com"; port="25"; delay="1"
##################################################################################
#
# Add usage function and help <<<<<<<<<<<< To Do
#
declare -A recipients # We are going to use associative array keys (and not values)
declare -A toList # to keep track of mailing lists ${!array[@]} as a builtin
declare -A ccList # way of preventing duplicates without sorting.
#
# add getopts options to handle "-x" type arguments. <<<<<<<<<<<< To Do
#
# grab remaining arguments (parse remaining $@ into newline delimited commands list) <<<<<<<<<<<< To Do
#
# read in commands from stdin between "$header" lines, if any
read -r line
if [[ "$line" == "$header" ]]; then
while read line && [[ "$line" != "$header" ]]; do commands+="$line"$'\n'; done
else
firstLine="$line" # Oops! Not a header afterall. Manaully add it to top of body later.
fi
#
# Process commands from whatever source (options, arguments, body header)
# usage: command:value1{:value2:value3}
numA=-1 # attachment numbering starts at 0
oIFS="$IFS"; IFS=$'\n'; for var in $commands; do
if [[ "$var" == *:* ]]; then # does it look like a command or a naked email?
IFS=":" read -r comm val1 val2 val3 < <(echo "$var") # using named FIFO instead of pipe to
# keep read variables in the current shell.
case "${comm,,}" in # usage: match lowercase version of comm
"subject"|"s") subject="$val1" ;;
"from"|"f") sender="$val1" ;;
"to"|"t") recipients["$val1"]=1; toList["$val1"]=1 ;;
"cc"|"c") recipients["$val1"]=1; ccList["$val1"]=1 ;;
"bcc"|"b") recipients["$val1"]=1 ;;
"attach"|"a") # usage: attach:att_path:att_MIME_type(optional):att_name(optional)
attachment[$((++numA))]="$val1"
attachmentType[numA]="$val2"
[[ "$val3" ]] && attachmentName[numA]="$val3" || attachmentName[numA]="${val1##*/}"
# if no name given, default to short filename from path
# add code for guessing MIME type from file extension if blank <<<<<<<<<<<< To Do
;;
"server") # usage: server(optional):server:port(optional):delay(optional)
[[ "$val1" ]] && server="$val1"
[[ "$val2" ]] && port="$val2"
[[ "$val3" ]] && delay="$val3"
;;
"port") port="$val1" ;;
"delay") delay="$val1" ;;
"host") host="$val1" ;;
"boundary") boundary="$val1" ;;
"warning") warning="$val1" ;;
"date") mailDate="$val1" ;;
*) echo "bad command"; exit 1 ;; #add usage/help here. <<<<<<<<<<<< To Do
esac
else recipients["$var"]=1; [[ ! "$blind" ]] && toList["$var"]=1
# naked email addresses are To: unless blind is set (then BCC:)
fi
done; IFS="$oIFS"
#
# Telnet Session
{ echo "HELO ""$host"
sleep "$delay"
echo "MAIL FROM: ""$sender"
sleep "$delay"
oIFS="$IFS"; IFS=$'\n'; for i in "${!recipients[@]}"; do
echo "RCPT TO: ""$i"
sleep "$delay"
done; IFS="$oIFS"
echo "DATA"
sleep "$delay"
echo "From: ""$sender"
toString="To: "
oIFS="$IFS"; IFS=$'\n'; for i in "${!toList[@]}"; do
toString+="$i"", "; done; IFS="$oIFS" # comma+space delimited
echo "${toString%, }" # remove last comma+spave
ccString="CC: "
oIFS="$IFS"; IFS=$'\n'; for i in "${!ccList[@]}"; do ccString+="$i"", "; done; IFS="$oIFS"
echo "${ccString%, }"
echo "Date: ""$mailDate"
echo "Subject: ""$subject"
echo "Content-Type: multipart/mixed; boundary=\"""$boundary""\""
echo "MIME-Version: 1.0"
echo
echo "$warning"
echo
echo "--""$boundary"
echo "Content-Type: text/plain"
echo
if [[ "$firstLine" ]]; then echo "$firstLine"; fi
while read line && [[ "$line" != "$header" ]]; do echo "$line"; done
echo
for i in `seq 0 "$numA"`; do
echo "--""$boundary"
echo "Content-Type: ""{$attachmentType[$i]}""; name=\"""${attachmentName[$i]}""\""
echo "Content-Transfer-Encoding: base64";
echo "Content-Disposition: attachment; filename=\"""${attachmentName[$i]}""\""
echo
base64 <"${attachment[$i]}"
echo
done
echo "--""$boundary""--"
echo "."
sleep "$delay"
echo "QUIT"; } | telnet "$server" "$port"
#
exit 0
Yes I did consider single quotes but thought the implementation added nothing interesting and detracted from the elegance of the example so consider that an "exercise for the reader". Backslash support is quite interesting, and as read only consumes single characters, it can be achieved with a single line case clause.
I didn't consider multiple white space, and this requires another state variable:
function quoted_read
{
typeset lInStr=0
typeset chBuffer
typeset chAct
typeset PrevWS=1
[ $# -eq 0 ] && set -- REPLY
OIFS=$IFS
IFS=
while read -N 1 -r chAct
do
case "$chAct" in
\") (( lInStr = (lInStr + 1) % 2 ))
continue
;;
\\) read -N 1 chAct ;;
" "|$'\n'|$'\t')
if [ $lInStr -eq 0 ]
then
[ "$chAct" == $'\n' ] && break
[ $PrevWS -ne 0 ] && continue
if [ $# -gt 1 ]
then
# More vars to assign -> not appending to current
export $1="$chBuffer"
shift
chBuffer=""
PrevWS=1
continue
fi
fi
;;
*) PrevWS=0 ;;
esac
chBuffer="${chBuffer}${chAct}"
done
export $1="$chBuffer"
shift
# Blank any vars not assigned
while [ $# -gt 0 ]
do
export $1=""
shift
done
IFS=$OIFS
[ "$chAct" == $'\n' ]
}
In relation to random string generation: if you have openssl you could also use:
boundary="$(openssl rand -hex 32)"
The way i wrote it the (unquoted) blank is a "terminal" (an expression ending the current parsing entity) - several consecutive terminals produce empty parsing entities, unless explicitly suppressed. You can do that in the " " part of the case-construct by inserting something like:
" ")
if [ -n "$chBuffer" ] ; then
# output/process the buffer which just ended
else
# do nothing
fi
;;
As for a little theory behind all that: i suggest the "Dragon Book": "Principles of Compiler Design"; Sethi, Aho, Ullmann. IMHO a must for the library of any programmer. It should be the third book apprentices buy, after the TAOCP by Knuth and "The C Programming Language" by Kernighan/Ritchie.
It is a - rather common - misconception that quoting can in any way be nested. I can not! In fact the shell works the same way as my parser: it maintains a flag "inside/outside quoted string" which is switched upon every occurrence of a quote (bar escaping, etc.). So "...."...."....." is read: inside quotes after first, outside after second, inside after third and outside again after fourth quote char.
A single quote inside a double-quoted string (and vice versa) is treated as a normal character without any special meaning.
I hope this helps.
bakunin