Url encoding a string using sed

Shell Programming and Scripting
1

Hi I was hoping some one would know if it is possible to url encode a string using sed?
My problem is I have extracted some key value pairs from a text file with sed, and will be inserting these pairs as source variables into a curl script to automatically download some xml from our server.
My problem is that one of the value pairs needs to be url encoded. The "j_password=0NRx5KpiquE32LrlXApzrA==" needs to be url encoded
Below are my key value pairs

JSESSIONID0=1omyjon28ge2p8dnuo0ngrjps
j_password=0NRx5KpiquE32LrlXApzrA==
iv=e10d17fd73b1f684986ecc3afbe68ff3
salt=90c6ea71b949661dcfb4b6ccaf51ecc6
_uc=MDSz8K74i642sedqmp88Jg==
_ut=1518796880482
JSESSIONID=1ozzynzzoqu8o1sgbqhuoek5hy
SSO_ID=5ohshkk9is9eb
workstationId=nfFqyQ0vDc2Bs5IQdhZNQg==

These pairs will be read into a curl script, Below is part of the line from my curl script that the key value pairs will be read in as variables;

 curl 'http://192.168.1.55/epace/do-login'  lastLoggedInCompany=public;~~ JSESSIONID=$JSESSIONID0' -H 'Origin: http://192.168.1.55' -H~~'Referer: http://192.168.1.55/epace/login.html' -H 'Connection: keep-alive' --data 'j_username=ticorder&j_password=$j_password&lastLoggedInCompany=public&iv=$iv&salt=$salt&hash=' --compressed

I had read online that curl can be used to url encode. Although I am not sure how I would use this function in curl to encode only part of the curl command that I need url encoded?
The script I am using to extract the value pairs is below

sed  -i ' -e ' -n '/POST/,$p' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e $'s/\t/\ /g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e $'s/\ \ /\ /g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e $'s/\ GET.*//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e $'s/.*_uc\=/_uc\=/g' /WebLoads/MainGo/CaptureFile-0
sed -i ' -e ' -n '/.*/,// G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e $'s/\t/\ /g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/.*\=public\;\ JSESSIONID\=/JSESSIONID0\=/g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/Form item\:\ /\'$'\n/g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/\ POST.*//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/.*j_username.*//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/.*companySelection.*//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/.*Login.*//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/.*lastLoggedInCompany.*//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/.*hash.*//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/\;\ /\'$'\n/g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/\"\ =\ \"/\=/g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/\"\,//g' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e s'/^\"//g' /WebLoads/MainGo/CaptureFile-0
sed -i ' -e ' -n '/.*/,// G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P' /WebLoads/MainGo/CaptureFile-0
sed -i '' -e '/^\s*$/d' /WebLoads/MainGo/CaptureFile-0

I've looked online and it does not seem very hopeful to be able to accomplish this with sed, but It would be my first choice since my understanding of sed is better than some of the other available programs. If there is another way of accomplishing this with another program I would be more than thankful if some one was able to show me how this could be done.
Thank you
Paul

2

You could use a shell function to encode a passed string into a variable like this:

#!/bin/bash

url_var() {
  # URL encode string and optionally store in shell variable
  #
  # usage: url_var <string> [var]

  local LC_ALL=C
  local encoded=""
  local i c

  for (( i=0 ; i<${#1} ; i++ )); do
     c=${1:$i:1}
     case "$c" in
        [a-zA-Z0-9/_.~-] ) encoded="${encoded}$c" ;;
        * ) printf -v encoded "%s%%%02x" "$encoded" "'${c}" ;;
     esac
  done
  [ $# -gt 1 ] &&
     printf -v "$2" "%s" "${encoded}" ||
     printf "%s\n" "${encoded}"
}

url_var j_password=0NRx5KpiquE32LrlXApzrA== PASS
printf "PASS=%s\n" "$PASS"
3

I'm not sure your curl command does what you intend it to do, as e.g. the pairs of single quotes don't match up. Once you sourced your "key / value pair" file so all the needed shell variables are defined, use Chubler_XL's function (in a "command substitution"!) to encode the variables in question:

curl 'http://192.168.1.55/epace/do-login  lastLoggedInCompany=public;~~ JSESSIONID='$JSESSIONID0' -H Origin: http://192.168.1.55 -H~~Referer: http://192.168.1.55/epace/login.html -H Connection: keep-alive --data j_username=ticorder&j_password='$(url_var $j_password)'&lastLoggedInCompany=public&iv='$iv'&salt='$salt'&hash= --compressed'

Not sure if I entirely and correctly understood your request, nor if the other variables used wouldn't need a URL-encoding as well.

4

Hi thank you for helping me with my problem, I appreciate the time you've taken very much.
The curl command I am using is one that I have used before to download our needed xml from our server. It has stopped working due to an upgrade on the server it was logging into.
The server we were extracting the xml from is now using more dynamic security that requires a new password and supporting cookies every twenty four hours, I was able to extract the key value pairs using tshark from network trafic and had hoped to insert these into my curl command as variables, it does seem strange that the only one that requires url encoding is the j_pasword, I am just following the format of the existing command which url encodes this for some reason.
I has hoping you could go into some more detail on how to use Chubler_XL shell function and how it works as I would like to learn for myself what it is doing.
Should I save this as a file and make a call to it?
Thank you
Paul

5

You can take above as an example and encode every single variable that you use in your command. like $(url_var $JSESSIONID0) etc. Give it a try and report back.

1 Like
6

Hi thanks for getting me going with this it seems to be working for me a little better for now.
I had never used a shell function before and this is my first exposure to one.
Ill try and puzzle out any remaining problems .

Thanks for all the help!!