# URL encoding

**URL:** <https://community.unix.com/t/url-encoding/191916>\
**Category:** Shell Programming and Scripting\
**Created:** [April 6, 2008, 7:30am UTC](https://community.unix.com/t/url-encoding/191916 "2008-04-06T07:30:10Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vichu](https://community.unix.com/letter_avatar/vichu/32/5_5575768a8748004e209b776fc1b2916d.png) [@Vichu](https://community.unix.com/u/Vichu)\
**Post date:** [April 6, 2008, 7:30am UTC](https://community.unix.com/t/url-encoding/191916/1 "2008-04-06T07:30:10Z")

</div>

Hi All,

I want to do URL encoding using shell script in my project. I decided that the sed is the correct tool to do this. But I am unable achieve what I wanted using sed. kindly help me to get rid of this.

My requirement is , there will be one URL with all special character, spaces etc...

For ex. [xxxxxx.com&nbsp;-&nbsp;xxx sex videos free hd porn Resources and Information.](https://www.xxxxxx.com/change&%24) ^this to?%checkthe@-functionality..I want to do URL encoding only after �?� mark. Final result should be [xxxxxx.com&nbsp;-&nbsp;xxx sex videos free hd porn Resources and Information.](https://www.xxxxxx.com/change&%24) ^this to?\<encoded 2nd part\>

Thanks in advance

Regards  
Vichu

---

<div class="post-metadata">

**Author:** ![era](https://community.unix.com/letter_avatar/era/32/5_5575768a8748004e209b776fc1b2916d.png) [@era](https://community.unix.com/u/era)\
**Post date:** [April 6, 2008, 7:48am UTC](https://community.unix.com/t/url-encoding/191916/2 "2008-04-06T07:48:43Z")

</div>

There is nothing after the ? which requires encoding except the percent sign. Perhaps you should pick a more detailed example.

It's not going to be very elegant to do this in sed because it requires a loop, and loops are kind of tricky in sed. Basically, stash away the part you don't want to encode, loop over the remaining part, moving away everything you have already encoded by appending it to the stash.

Maybe something like this, instead?

```nohighlight
perl -ple 's/\?(.*)//;
my $tail = $1;
$tail =~ s/([%? +&!<>()])/sprintf "%%%02x", ord($1) /ge;
s/$/$tail/'

```

The list of characters which require or might benefit from escaping is quite probably not complete. This assumes you have nothing after the URL which is not part of the URL, and that the first question mark separates the tail which requires encoding from the base URL.

---

<div class="post-metadata">

**Author:** ![Vichu](https://community.unix.com/letter_avatar/vichu/32/5_5575768a8748004e209b776fc1b2916d.png) [@Vichu](https://community.unix.com/u/Vichu)\
**Post date:** [April 6, 2008, 8:03am UTC](https://community.unix.com/t/url-encoding/191916/3 "2008-04-06T08:03:05Z")

</div>

Seperating URL into two parts (? is the delimiter before ? is the first part and after ? is the second part) and I can do encoding for the second part using awk..But I don't want to split the line. Just skipping first part of URL and encoding the second part.

---

<div class="post-metadata">

**Author:** ![era](https://community.unix.com/letter_avatar/era/32/5_5575768a8748004e209b776fc1b2916d.png) [@era](https://community.unix.com/u/era)\
**Post date:** [April 6, 2008, 8:36am UTC](https://community.unix.com/t/url-encoding/191916/4 "2008-04-06T08:36:53Z")

</div>

Did you try the code I posted?

---

<div class="post-metadata">

**Author:** ![Vichu](https://community.unix.com/letter_avatar/vichu/32/5_5575768a8748004e209b776fc1b2916d.png) [@Vichu](https://community.unix.com/u/Vichu)\
**Post date:** [April 7, 2008, 1:11am UTC](https://community.unix.com/t/url-encoding/191916/5 "2008-04-07T01:11:53Z")

</div>

Hi era,

Thanks for your promptness..But I don't want to do it in perl...My requirement is in shell..

---

<div class="post-metadata">

**Author:** ![V3l0](https://community.unix.com/letter_avatar/v3l0/32/5_5575768a8748004e209b776fc1b2916d.png) [@V3l0](https://community.unix.com/u/V3l0)\
**Post date:** [April 7, 2008, 5:16am UTC](https://community.unix.com/t/url-encoding/191916/6 "2008-04-07T05:16:30Z")

</div>

Use 2 files:  
urlencode.sed

```nohighlight
s/%/%25/g
s/ /%20/g
s/ /%09/g
s/!/%21/g
s/"/%22/g
s/#/%23/g
s/\$/%24/g
s/\&/%26/g
s/'\''/%27/g
s/(/%28/g
s/)/%29/g
s/\*/%2a/g
s/+/%2b/g
s/,/%2c/g
s/-/%2d/g
s/\./%2e/g
s/\//%2f/g
s/:/%3a/g
s/;/%3b/g
s//%3e/g
s/?/%3f/g
s/@/%40/g
s/\[/%5b/g
s/\\/%5c/g
s/\]/%5d/g
s/\^/%5e/g
s/_/%5f/g
s/`/%60/g
s/{/%7b/g
s/|/%7c/g
s/}/%7d/g
s/~/%7e/g
s/ /%09/g

```

urlencode.sh

```nohighlight
#!/bin/ksh

STR1=$(echo "https://www.xxxxxx.com/change&$ ^this to?%checkthe@-functionality" | cut -d\? -f1)
STR2=$(echo "https://www.xxxxxx.com/change&$ ^this to?%checkthe@-functionality" | cut -d\? -f2)

OUT2=$(echo "$STR2" | sed -f urlencode.sed)

echo "$STR1?$OUT2"

```

Result:

```nohighlight
./urlencode.sh
https://www.xxxxxx.com/change&$ ^this to?%25checkthe%40%2dfunctionality

```

OK for you ? 🆒

---

<div class="post-metadata">

**Author:** ![Vichu](https://community.unix.com/letter_avatar/vichu/32/5_5575768a8748004e209b776fc1b2916d.png) [@Vichu](https://community.unix.com/u/Vichu)\
**Post date:** [April 7, 2008, 6:06am UTC](https://community.unix.com/t/url-encoding/191916/7 "2008-04-07T06:06:05Z")

</div>

I proposed the same procedure,whatever you have given here, using awk.But my team didnot accept it. They dont want to use extra varialbes.They would like to do encoding (IInd part) on the same variable.

# My proposed steps

str1=`echo "[https://www.xxxxxx.com/change&$](https://www.xxxxxx.com/change&$) ^this to?%checkthe@-functionality" | awk -F? '{print $1}'`

str2=`echo "[https://www.xxxxxx.com/change&$](https://www.xxxxxx.com/change&$) ^this to?%checkthe@-functionality" | awk -F? '{print $2}'`

str3=`echo $str2 | sed -f ./seq.sed `

echo $str?$str3

Do you have any idea?

---

<div class="post-metadata">

**Author:** ![V3l0](https://community.unix.com/letter_avatar/v3l0/32/5_5575768a8748004e209b776fc1b2916d.png) [@V3l0](https://community.unix.com/u/V3l0)\
**Post date:** [April 7, 2008, 11:03am UTC](https://community.unix.com/t/url-encoding/191916/8 "2008-04-07T11:03:43Z")

</div>

OK, in one file :

```nohighlight
#!/bin/ksh

ORIG="https://www.xxxxxx.com/change&$ ^this to?%checkthe@-functionality"
STR1=$(echo $ORIG | cut -d\? -f1)
STR2=$(echo $ORIG | cut -d\? -f2)

OUT=$STR2

FORMULA="s/%/%25/g=s/ /%20/g=s/ /%09/g=s/!/%21/g=s/\"/%22/g=s/#/%23/g=s/\\\\$/%24/g=s/\&/%26/g"
FORMULA="${FORMULA}=s/'\''/%27/g=s/(/%28/g=s/)/%29/g=s/\*/%2a/g=s/+/%2b/g=s/,/%2c/g=s/-/%2d/g"
FORMULA="${FORMULA}=s/\./%2e/g=s/\//%2f/g=s/:/%3a/g=s/;/%3b/g=s/?/%3f/g=s/@/%40/g"
FORMULA="${FORMULA}=s/\\\\\\/%5b/g=s/\\\\\\\\/%5c/g=s/\]/%5d/g=s/\^/%5e/g=s/_/%5f/g=s/\`/%60/g=s/{/%7b/g"
FORMULA="${FORMULA}=s/|/%7c/g=s/}/%7d/g=s/~/%7e/g=s/ /%09/g"

i=1

while [i]
do
        z=$(echo "$FORMULA" | cut -d= -f$i)

        if [! x"$z" = x]
        then
                y=$(echo $OUT | sed -e "$z")
                OUT=$y
        else
                break
        fi
        let i=$i+1
done

echo Original: $ORIG
echo URL encoded after '?': "$STR1?$OUT"

```

Result

```nohighlight
./urlencode.sh
Original: https://www.xxxxxx.com/change&$ ^this to?%checkthe@-functionality
URL encoded after ?: https://www.xxxxxx.com/change&$ ^this to?%25checkthe%40%2dfunctionality

```

:b::b:😕😕

---

<div class="post-metadata">

**Author:** ![wozza.xing](https://community.unix.com/user_avatar/community.unix.com/wozza.xing/32/4431_2.png) [@wozza.xing](https://community.unix.com/u/wozza.xing)\
**Post date:** [August 27, 2008, 7:16pm UTC](https://community.unix.com/t/url-encoding/191916/9 "2008-08-27T19:16:56Z")

</div>

echo -e $(echo %23 | sed -e "s/%\([0-9a-f]\{2\}\)/\\\x\1/")