# Problem in tokenizing the string

**URL:** <https://community.unix.com/t/problem-in-tokenizing-the-string/309191>\
**Category:** Shell Programming and Scripting\
**Created:** [April 27, 2012, 2:09am UTC](https://community.unix.com/t/problem-in-tokenizing-the-string/309191 "2012-04-27T02:09:24Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![saurabhkoar](https://community.unix.com/letter_avatar/saurabhkoar/32/6_449c533f586939b6e5f246632a46421f.png) [@saurabhkoar](https://community.unix.com/u/saurabhkoar)\
**Post date:** [April 27, 2012, 2:09am UTC](https://community.unix.com/t/problem-in-tokenizing-the-string/309191/1 "2012-04-27T02:09:24Z")

</div>

Supposed I have a string in the format:

```nohighlight
<service_name> = <ldap user FDN> : <password>

```

like

```nohighlight
DNS = cn=user1,o=company : pwd

```

I want to tokenize like:

```nohighlight
service name:DNS
UserName: cn=user1,o=company
Password: pwd.

```

Because of the '=' sign between Service Name and LDAP Name I not struggling to parse it. Please help.

---

<div class="post-metadata">

**Author:** ![itkamaraj](https://community.unix.com/user_avatar/community.unix.com/itkamaraj/32/1880_2.png) [@itkamaraj](https://community.unix.com/u/itkamaraj)\
**Post date:** [April 27, 2012, 2:24am UTC](https://community.unix.com/t/problem-in-tokenizing-the-string/309191/2 "2012-04-27T02:24:25Z")

</div>

```nohighlight
$ echo "DNS = cn=user1,o=company : pwd" | nawk -F"[\=:]" '{print "Service Name : " $1; print "Password : " $NF}{sub($1"=","");sub(":"$NF,"");print "UserName : " $0}'
Service Name : DNS 
Password : pwd
UserName : cn=user1,o=company 

```

---

<div class="post-metadata">

**Author:** ![saurabhkoar](https://community.unix.com/letter_avatar/saurabhkoar/32/6_449c533f586939b6e5f246632a46421f.png) [@saurabhkoar](https://community.unix.com/u/saurabhkoar)\
**Post date:** [April 27, 2012, 2:52am UTC](https://community.unix.com/t/problem-in-tokenizing-the-string/309191/3 "2012-04-27T02:52:56Z")

</div>

Actually I don't want to print in that format. I want the corresponding variables variables assigned to the corresponding value. As I will need the variables further. Like I want servicename variable assigned to DNS. So, if I do echo $servicename, it will print DNS. If I do echo $password it will print pwd, if I do echo $username it will print cn=user1,o=company.

So, I did something like this:

```nohighlight
echo "DNS = cn=user1,o=company : pwd" | awk -F"[=:]" '{servicename="$1"; password="$NF"}{sub($1"=","");sub(":"$NF,"");Username="$0"}'

```

And then tried echo. But it didn't print anything. Please correct me where I am wrong.

---

<div class="post-metadata">

**Author:** ![saurabhkoar](https://community.unix.com/letter_avatar/saurabhkoar/32/6_449c533f586939b6e5f246632a46421f.png) [@saurabhkoar](https://community.unix.com/u/saurabhkoar)\
**Post date:** [April 28, 2012, 6:35am UTC](https://community.unix.com/t/problem-in-tokenizing-the-string/309191/4 "2012-04-28T06:35:43Z")

</div>

Any response on this?

---

<div class="post-metadata">

**Author:** ![drl](https://community.unix.com/user_avatar/community.unix.com/drl/32/542_2.png) [@drl](https://community.unix.com/u/drl)\
**Post date:** [April 28, 2012, 11:18am UTC](https://community.unix.com/t/problem-in-tokenizing-the-string/309191/5 "2012-04-28T11:18:14Z")

</div>

Hi.

Edit -- I didn't notice the embedded "=" sign, so the solution from neutronscott, q.v., is the correct one for using bash built-ins.

With bash shell built-ins:

```nohighlight
#!/usr/bin/env bash

# @(#) s1	Demonstrate parse line into variables, bash built-in read.

# List environment for demonstration computation.
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [-f $C] && $C

line="DNS = cn=user1,o=company : pwd"
pl " Line to be parsed into variables:"
pe "$line"

oldifs="$IFS"
IFS="=:" read s u p junk <<< "$line"

pl " Results of setting variables from read:"
pe "service name: $s"
pe "UserName: $u"
pe "Password: $p"

pl " Results with all spaces eliminated with \"Parameter Expansion\":"
t1=${line// /}
pe " t1 is \"$t1\""
IFS="=:" read s u p junk <<< "$t1"
pe "service name: $s"
pe "UserName: $u"
pe "Password: $p"

IFS="$oldifs"

exit 0

```

producing:

```nohighlight
% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution : Debian GNU/Linux 5.0.8 (lenny) 
bash GNU bash 3.2.39

-----
 Line to be parsed into variables:
DNS = cn=user1,o=company : pwd

-----
 Results of setting variables from read:
service name: DNS 
UserName: cn
Password: user1,o

-----
 Results with all spaces eliminated with "Parameter Expansion":
 t1 is "DNS=cn=user1,o=company:pwd"
service name: DNS
UserName: cn
Password: user1,o

```

See man bash for details.

Best wishes ... cheers, drl

---

<div class="post-metadata">

**Author:** ![neutronscott](https://community.unix.com/user_avatar/community.unix.com/neutronscott/32/2331_2.png) [@neutronscott](https://community.unix.com/u/neutronscott)\
**Post date:** [April 28, 2012, 11:50am UTC](https://community.unix.com/t/problem-in-tokenizing-the-string/309191/6 "2012-04-28T11:50:50Z")

</div>

```nohighlight
[mute@geek ~]$ bash ./test
[DNS] [cn=user1,o=company] [pwd]
[mute@geek ~]$ cat test
#!/bin/bash

var='DNS = cn=user1,o=company : pwd'

service=${var%% *}
user=${var#* = }
pass=${user#* : }
user=${user% : *}

echo "[$service] [$user] [$pass]"

```

---

<div class="post-metadata">

**Author:** ![saurabhkoar](https://community.unix.com/letter_avatar/saurabhkoar/32/6_449c533f586939b6e5f246632a46421f.png) [@saurabhkoar](https://community.unix.com/u/saurabhkoar)\
**Post date:** [April 28, 2012, 3:41pm UTC](https://community.unix.com/t/problem-in-tokenizing-the-string/309191/7 "2012-04-28T15:41:01Z")

</div>

Can you please explain the logic?

Also if I miss the blank between DNS and username then it doesn't work.

```nohighlight
wgp-fap-109:~ # ./srk.sh
DNS=cn=afpuser,o=company
DNS=cn=afpuser,o=company
pwd
wgp-fap-109:~ # cat srk.sh
#!/bin/bash

var='DNS=cn=afpuser,o=company : pwd'

service=${var%% *}
user=${var#* = }
pass=${user#* : }
user=${user% : *}

echo $service
echo $user
echo $pass

```

---

<div class="post-metadata">

**Author:** ![neutronscott](https://community.unix.com/user_avatar/community.unix.com/neutronscott/32/2331_2.png) [@neutronscott](https://community.unix.com/u/neutronscott)\
**Post date:** [April 28, 2012, 3:48pm UTC](https://community.unix.com/t/problem-in-tokenizing-the-string/309191/8 "2012-04-28T15:48:43Z")

</div>

# will remove string from beginning. % from the end. Double them, ## or %% to strip longest match..

So if your spaces are optional you need more work. If spaces won't be there, remove them from the right side.

Look at "Parameter Expansion" in shell manual

---

<div class="post-metadata">

**Author:** ![drl](https://community.unix.com/user_avatar/community.unix.com/drl/32/542_2.png) [@drl](https://community.unix.com/u/drl)\
**Post date:** [April 28, 2012, 10:58pm UTC](https://community.unix.com/t/problem-in-tokenizing-the-string/309191/9 "2012-04-28T22:58:01Z")

</div>

Hi.

Here is an amended script that follows the idea of using built-in read to parse the tokens. The difference is that spaces are deleted, then characters that separate the tokens are converted to underlines, "\_", and then that is used as the separator.

Here are two ways of doing that, one with sed, the other with parameter expansion:

```nohighlight
#!/usr/bin/env bash

# @(#) s2	Demonstrate parse line into variables, bash built-in read.

# List environment for demonstration computation.
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
db() { : ; }
C=$HOME/bin/context && [-f $C] && $C sed

line="DNS = cn=user1,o=company : pwd"
pl " Line to be parsed into variables:"
pe "$line"
line=$( echo "$line" | sed -e 's/ //g' -e 's/=/_/' -e 's/:/_/')
pe " Line prepared by sed:"
pe "$line"

oldifs="$IFS"
IFS="_" read s u p junk <<< "$line"

pl " Results of setting variables from sed, read:"
pe "service name: $s"
pe "UserName: $u"
pe "Password: $p"

unset s u p
pl " Sequence of \"Parameter Expansions\":"
line="DNS = cn=user1,o=company : pwd"
pe "$line"
line=${line// /}
pe "$line"
line=${line/=/_}
pe "$line"
line=${line/:/_}
pe "$line"
IFS="_" read s u p junk <<< "$line"
pl " Results of setting variables from expansion, read:"
pe "service name: $s"
pe "UserName: $u"
pe "Password: $p"

IFS="$oldifs"

exit 0

```

producing:

```nohighlight
% ./s2

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution : Debian GNU/Linux 5.0.8 (lenny) 
bash GNU bash 3.2.39
sed GNU sed version 4.1.5

-----
 Line to be parsed into variables:
DNS = cn=user1,o=company : pwd
 Line prepared by sed:
DNS_cn=user1,o=company_pwd

-----
 Results of setting variables from sed, read:
service name: DNS
UserName: cn=user1,o=company
Password: pwd

-----
 Sequence of "Parameter Expansions":
DNS = cn=user1,o=company : pwd
DNS=cn=user1,o=company:pwd
DNS_cn=user1,o=company:pwd
DNS_cn=user1,o=company_pwd

-----
 Results of setting variables from expansion, read:
service name: DNS
UserName: cn=user1,o=company
Password: pwd

```

Best wishes ... cheers, drl