Using sed with special characters produced from crypto

UNIX for Beginners Q & A UNIX for Dummies Questions & Answers
1

Hey there,

I'm facing some weird issues with sed when trying to do substitution in a text file with the content of some environment variables. Those variables are used to store crypted (3DES) info with much special characters and that's where the problem starts.

I've already tried to use both weak and strong quotes to protect the content of the variables (and \ to protect characters) passed to sed but I still get truncated values injected instead of the whole thing :S (well actually I've identified the $!@^� character in cause and guess what it seems like it's a ETX character so end of text).

Has anyone a clue or a solution to get over this? :confused:

2

Please supply an example.

3

That'll have to way til monday. The ETX character was the last one I had to deal with before I stopped and I'm almost sure I'll face the same problem with some other control characters.

4

Raw encrypted data is going to be raw bytes without regard to representation. I don't think you can expect to keep such data in shell variables raw, NULLS and things which just can't be represented as an ASCII string will mess it up. Even if you somehow got them in, you'll have real trouble passing them to anything else.

In what way do they need to be used?

$ echo asdf > file
$ openssl base64 < file
YXNkZgo=
$ echo "YXNkZgo=" | openssl enc -d base64
asdf
$

You could pipe the data from a file into openssl base64 to get safe base64-encoded data(which is always representable in ASCII) to store in your shell variables, then get the binary data back later from it and pipe it where it needs to go.

5

Thanks for the reply Corona688, I'll give it a try.

I actually have to store that encrypted data in shell vars which looks like ok the way I did. As you stated above the problem is then to pass that data to other scripts as the non representable characters make sed go nuts :S

I just hope openssl is available on the system I'm working on.

Once again thank you for the hint :wink:

##########################
EDIT:

I've tried to encode / decode with base64 and it seems to work fine (100.000 auto generated values of 20 characters used in my tests). I can provide java code template for those who might be interested.

Thanks Apache for commons-codec-1.5.jar

6

Should be available on nearly all Linux systems. Other places may have different commands to handle base64.

7

Well I'm working on a AIX system with much restrictions so I'm not so sure it'll be available. Anyway I worked around it with some apache lib. Thx again for the hint :wink:

8

Yep, openssl is available since AIX 5.3 either on the base OS media or maybe on any expansion pack media.