Another binary manipulation thread.

As bash cannot cope with a byte value of zero inside a variable then this is a workaround.

This code is a DEMO to show how to create a _string_variable_ containing all of the values of 1 to 255 as single characters and 0, (zero), as a two byte character set of \0.
The real binary files are 512 bytes in size and contains 2 zeros.
The _string_variable_ is 514 bytes in size and contains \0 for the byte value zero.

Ignore the simplicity of the for loops to generate the binary files it is the 'text' variable that is ultimately important.

I am not sure if it is of any use but shows that the bash shell can cope with binary...

#!/bin/bash --posix

# Creating a binary variable from a binary file...

> /tmp/SomeBinaryFile.dat
> /tmp/BinData.dat

# This loop is to give a usable 512 byte binary file only from 0x00 to 0xFF continuous repeated twice...
clear
echo "Generate a 512 byte binary file..."
echo ""
for character in $( seq 0 1 255 )
do
	# Note:- The backticks and the four escape characters ARE required...
	char=`printf '\\\\x'"%02x" $character`
	printf "$char" >> /tmp/SomeBinaryfile.dat
done
for character in $( seq 0 1 255 )
do
	# Do a repeat of the last loop, copy-n-paste is SOOO easy... ;o)
	char=`printf '\\\\x'"%02x" $character`
	printf "$char" >> /tmp/SomeBinaryfile.dat
done

# Now display the file information...
ls -l /tmp/SomeBinaryFile.dat

# Now do a HEX dump as proof...
echo ""
hexdump -C /tmp/SomeBinaryFile.dat
echo "" 

# We are now going to create a variable with pseudo-zeros inside it. All other 8 bit values will eventually be real.
character=""
for subscript in $( seq 0 1 255 )
do
	hexdata=`hexdump -n1 -s$subscript -v -e '1/1 "%02x"' /tmp/SomeBinaryFile.dat`
	char=`printf '\\\\x'"$hexdata"`
	if [ "$hexdata" == "00" ]
	then
		character=$character'\\0'
	fi
	character="$character$char"
done
# Repeat the above, copy-n-paste again... ;o)
for subscript in $( seq 256 1 511 )
do
	hexdata=`hexdump -n1 -s$subscript -v -e '1/1 "%02x"' /tmp/SomeBinaryFile.dat`
	char=`printf '\\\\x'"$hexdata"`
	if [ "$hexdata" == "00" ]
	then
		# Note:- Two backslashes required here to generate the \0 reqired for the variable.
		character=$character'\\0'
	fi
	character=$character$char
done

# Now generate the variable 'text' filled with binary except 0, this is changed to \0 inside the variable.
text=`echo -e -n "$character"`
echo -e -n "$text" > /tmp/BinData.dat
hexdump -C /tmp/BinData.dat

# Proof run.
echo -e "$text"
echo ""
echo "Length of string variable is ${#text} characters, NOT 512 characters..."
echo ""
echo "NOTE:- Zero, (00), is missing and replaced by real text '\0', as a"
echo "pseudo-zero. All other byte values exist..."
echo ""

Here is the display on a Macbook Pro 13" OSX 10.7.5 using the default Terminal.

Last login: Tue Aug 13 20:30:47 on ttys000
AMIGA:barrywalker~> ./bin_variable.sh

Generate a 512 byte binary file...

-rw-r--r--  1 barrywalker  wheel  512 13 Aug 21:19 /tmp/SomeBinaryFile.dat

00000000  00 01 02 03 04 05 06 07  08 09 0a 0b 0c 0d 0e 0f  |................|
00000010  10 11 12 13 14 15 16 17  18 19 1a 1b 1c 1d 1e 1f  |................|
00000020  20 21 22 23 24 25 26 27  28 29 2a 2b 2c 2d 2e 2f  | !"#$%&'()*+,-./|
00000030  30 31 32 33 34 35 36 37  38 39 3a 3b 3c 3d 3e 3f  |0123456789:;<=>?|
00000040  40 41 42 43 44 45 46 47  48 49 4a 4b 4c 4d 4e 4f  |@ABCDEFGHIJKLMNO|
00000050  50 51 52 53 54 55 56 57  58 59 5a 5b 5c 5d 5e 5f  |PQRSTUVWXYZ[\]^_|
00000060  60 61 62 63 64 65 66 67  68 69 6a 6b 6c 6d 6e 6f  |`abcdefghijklmno|
00000070  70 71 72 73 74 75 76 77  78 79 7a 7b 7c 7d 7e 7f  |pqrstuvwxyz{|}~.|
00000080  80 81 82 83 84 85 86 87  88 89 8a 8b 8c 8d 8e 8f  |................|
00000090  90 91 92 93 94 95 96 97  98 99 9a 9b 9c 9d 9e 9f  |................|
000000a0  a0 a1 a2 a3 a4 a5 a6 a7  a8 a9 aa ab ac ad ae af  |................|
000000b0  b0 b1 b2 b3 b4 b5 b6 b7  b8 b9 ba bb bc bd be bf  |................|
000000c0  c0 c1 c2 c3 c4 c5 c6 c7  c8 c9 ca cb cc cd ce cf  |................|
000000d0  d0 d1 d2 d3 d4 d5 d6 d7  d8 d9 da db dc dd de df  |................|
000000e0  e0 e1 e2 e3 e4 e5 e6 e7  e8 e9 ea eb ec ed ee ef  |................|
000000f0  f0 f1 f2 f3 f4 f5 f6 f7  f8 f9 fa fb fc fd fe ff  |................|
00000100  00 01 02 03 04 05 06 07  08 09 0a 0b 0c 0d 0e 0f  |................|
00000110  10 11 12 13 14 15 16 17  18 19 1a 1b 1c 1d 1e 1f  |................|
00000120  20 21 22 23 24 25 26 27  28 29 2a 2b 2c 2d 2e 2f  | !"#$%&'()*+,-./|
00000130  30 31 32 33 34 35 36 37  38 39 3a 3b 3c 3d 3e 3f  |0123456789:;<=>?|
00000140  40 41 42 43 44 45 46 47  48 49 4a 4b 4c 4d 4e 4f  |@ABCDEFGHIJKLMNO|
00000150  50 51 52 53 54 55 56 57  58 59 5a 5b 5c 5d 5e 5f  |PQRSTUVWXYZ[\]^_|
00000160  60 61 62 63 64 65 66 67  68 69 6a 6b 6c 6d 6e 6f  |`abcdefghijklmno|
00000170  70 71 72 73 74 75 76 77  78 79 7a 7b 7c 7d 7e 7f  |pqrstuvwxyz{|}~.|
00000180  80 81 82 83 84 85 86 87  88 89 8a 8b 8c 8d 8e 8f  |................|
00000190  90 91 92 93 94 95 96 97  98 99 9a 9b 9c 9d 9e 9f  |................|
000001a0  a0 a1 a2 a3 a4 a5 a6 a7  a8 a9 aa ab ac ad ae af  |................|
000001b0  b0 b1 b2 b3 b4 b5 b6 b7  b8 b9 ba bb bc bd be bf  |................|
000001c0  c0 c1 c2 c3 c4 c5 c6 c7  c8 c9 ca cb cc cd ce cf  |................|
000001d0  d0 d1 d2 d3 d4 d5 d6 d7  d8 d9 da db dc dd de df  |................|
000001e0  e0 e1 e2 e3 e4 e5 e6 e7  e8 e9 ea eb ec ed ee ef  |................|
000001f0  f0 f1 f2 f3 f4 f5 f6 f7  f8 f9 fa fb fc fd fe ff  |................|
00000200

00000000  00 01 02 03 04 05 06 07  08 09 0a 0b 0c 0d 0e 0f  |................|
00000010  10 11 12 13 14 15 16 17  18 19 1a 1b 1c 1d 1e 1f  |................|
00000020  20 21 22 23 24 25 26 27  28 29 2a 2b 2c 2d 2e 2f  | !"#$%&'()*+,-./|
00000030  30 31 32 33 34 35 36 37  38 39 3a 3b 3c 3d 3e 3f  |0123456789:;<=>?|
00000040  40 41 42 43 44 45 46 47  48 49 4a 4b 4c 4d 4e 4f  |@ABCDEFGHIJKLMNO|
00000050  50 51 52 53 54 55 56 57  58 59 5a 5b 5c 5d 5e 5f  |PQRSTUVWXYZ[\]^_|
00000060  60 61 62 63 64 65 66 67  68 69 6a 6b 6c 6d 6e 6f  |`abcdefghijklmno|
00000070  70 71 72 73 74 75 76 77  78 79 7a 7b 7c 7d 7e 7f  |pqrstuvwxyz{|}~.|
00000080  80 81 82 83 84 85 86 87  88 89 8a 8b 8c 8d 8e 8f  |................|
00000090  90 91 92 93 94 95 96 97  98 99 9a 9b 9c 9d 9e 9f  |................|
000000a0  a0 a1 a2 a3 a4 a5 a6 a7  a8 a9 aa ab ac ad ae af  |................|
000000b0  b0 b1 b2 b3 b4 b5 b6 b7  b8 b9 ba bb bc bd be bf  |................|
000000c0  c0 c1 c2 c3 c4 c5 c6 c7  c8 c9 ca cb cc cd ce cf  |................|
000000d0  d0 d1 d2 d3 d4 d5 d6 d7  d8 d9 da db dc dd de df  |................|
000000e0  e0 e1 e2 e3 e4 e5 e6 e7  e8 e9 ea eb ec ed ee ef  |................|
000000f0  f0 f1 f2 f3 f4 f5 f6 f7  f8 f9 fa fb fc fd fe ff  |................|
00000100  00 01 02 03 04 05 06 07  08 09 0a 0b 0c 0d 0e 0f  |................|
00000110  10 11 12 13 14 15 16 17  18 19 1a 1b 1c 1d 1e 1f  |................|
00000120  20 21 22 23 24 25 26 27  28 29 2a 2b 2c 2d 2e 2f  | !"#$%&'()*+,-./|
00000130  30 31 32 33 34 35 36 37  38 39 3a 3b 3c 3d 3e 3f  |0123456789:;<=>?|
00000140  40 41 42 43 44 45 46 47  48 49 4a 4b 4c 4d 4e 4f  |@ABCDEFGHIJKLMNO|
00000150  50 51 52 53 54 55 56 57  58 59 5a 5b 5c 5d 5e 5f  |PQRSTUVWXYZ[\]^_|
00000160  60 61 62 63 64 65 66 67  68 69 6a 6b 6c 6d 6e 6f  |`abcdefghijklmno|
00000170  70 71 72 73 74 75 76 77  78 79 7a 7b 7c 7d 7e 7f  |pqrstuvwxyz{|}~.|
00000180  80 81 82 83 84 85 86 87  88 89 8a 8b 8c 8d 8e 8f  |................|
00000190  90 91 92 93 94 95 96 97  98 99 9a 9b 9c 9d 9e 9f  |................|
000001a0  a0 a1 a2 a3 a4 a5 a6 a7  a8 a9 aa ab ac ad ae af  |................|
000001b0  b0 b1 b2 b3 b4 b5 b6 b7  b8 b9 ba bb bc bd be bf  |................|
000001c0  c0 c1 c2 c3 c4 c5 c6 c7  c8 c9 ca cb cc cd ce cf  |................|
000001d0  d0 d1 d2 d3 d4 d5 d6 d7  d8 d9 da db dc dd de df  |................|
000001e0  e0 e1 e2 e3 e4 e5 e6 e7  e8 e9 ea eb ec ed ee ef  |................|
000001f0  f0 f1 f2 f3 f4 f5 f6 f7  f8 f9 fa fb fc fd fe ff  |................|
00000200
	


!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????


!"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????

Length of string variable is 514 characters, NOT 512 characters...

NOTE:- Zero, (00), is missing and replaced by real text '\0', as a
pseudo-zero. All other byte values exist...

AMIGA:barrywalker~> _

The bash shell can deal with binary output, sure, but binary input?

Hi DGPickett...

I assume you mean from keyboard input...

Last login: Tue Aug 20 21:18:52 on ttys000
AMIGA:barrywalker~> > /tmp/bin.dat
AMIGA:barrywalker~> read -p "INPUT:- " -e text ; echo -e -n "$text" > /tmp/bin.dat
INPUT:- \\\xFF\\\x00\\\xFF\\\x00
AMIGA:barrywalker~> hexdump -C /tmp/bin.dat
00000000  ff 00 ff 00                                       |....|
00000004
AMIGA:barrywalker~> _

The tty may be cooking up some binary input, but this is far short of handlong nulls -- they just passed through. Bash did not detect them as null.

Hi DGPickett...

I appreciate that the _string_ is not an actual character zero, but the original code replaces character zero with two characters - "\0".

I am on holiday/vacation ATM so gimme a bit of time to INPUT to a variable with the two pseudo-zero characters for an actual character zero and 0xFF as a single byte, a total of three characters...

If I get stuck I will certainly admit it... ;o)

---------- Post updated 21-08-13 at 09:52 AM ---------- Previous update was 20-08-13 at 10:29 PM ----------

Hi DGPickett...

Slightly bigger than 3 bytes... ;o)

This generates a variable "text" 10 bytes long containing three "\0" pseudo-zeros and other non-ascii characters. The binary file generated is either 7 or 10 bytes in size......

Last login: Wed Aug 21 08:53:01 on ttys000
AMIGA:barrywalker~> read -p "INPUT:- " character
INPUT:- \\\xFF\\\\0\\\x7F\\\\0\\\x80\\\\0\\\xFF
AMIGA:barrywalker~> text=`echo -e -n "$character"`
AMIGA:barrywalker~> echo -n "$text" > /tmp/bin.dat
AMIGA:barrywalker~> hexdump -C /tmp/bin.dat
00000000  ff 5c 30 7f 5c 30 80 5c  30 ff                    |.\0.\0.\0.|
0000000a
AMIGA:barrywalker~> echo -e -n "$text" > /tmp/bin.dat
AMIGA:barrywalker~> hexdump -C /tmp/bin.dat
00000000  ff 00 7f 00 80 00 ff                              |.......|
00000007
AMIGA:barrywalker~> # Real string length should be 10 bytes.
AMIGA:barrywalker~> echo "${#text}"
10
AMIGA:barrywalker~> _

This is the best workaround I can do WRT to byte value zero and reading from the keyboard.

Replied to DGPickett and just noticed the reply added itself as an EDIT to my previous upolad...

Sorry for any inconvenience...

Reading the bash man page, 'read -n 1' looks like a nice start. I am not sure what happens when you read a line feed or white space, so I guess I have to try it. I wrote all256 to put out bytes 0x00 through 0xff.

$ all256|while read -n 1 c
> do
>  echo read "'$c'"
> done
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read '
read ''
read ''
read '
      '
read '
      '
'ead '
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read '
read ''
read ''
read ''
read ''
read ''
read '!'
read '"'
read '#'
read '$'
read '%'
read '&'
read '''
read '('
read ')'
read '*'
read '+'
read ','
read '-'
read '.'
read '/'
read '0'
read '1'
read '2'
read '3'
read '4'
read '5'
read '6'
read '7'
read '8'
read '9'
read ':'
read ';'
read '<'
read '='
read '>'
read '?'
read '@'
read 'A'
read 'B'
read 'C'
read 'D'
read 'E'
read 'F'
read 'G'
read 'H'
read 'I'
read 'J'
read 'K'
read 'L'
read 'M'
read 'N'
read 'O'
read 'P'
read 'Q'
read 'R'
read 'S'
read 'T'
read 'U'
read 'V'
read 'W'
read 'X'
read 'Y'
read 'Z'
read '['
read ']'
read '^'
read '_'
read '`'
read 'a'
read 'b'
read 'c'
read 'd'
read 'e'
read 'f'
read 'g'
read 'h'
read 'i'
read 'j'
read 'k'
read 'l'
read 'm'
read 'n'
read 'o'
read 'p'
read 'q'
read 'r'
read 's'
read 't'
read 'u'
read 'v'
read 'w'
read 'x'
read 'y'
read 'z'
read '{'
read '|'
read '}'
read '~'
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ''
read ' '
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
read '�'
$ all256|while read -n 1 c; do  echo read "'$c'"; done|cat -vt
read ''
read '^A'
read '^B'
read '^C'
read '^D'
read '^E'
read '^F'
read '^G'
read '^H'
read ''
read ''
read '^K'
read '^L'
read '^M'
read '^N'
read '^O'
read '^P'
read '^Q'
read '^R'
read '^S'
read '^T'
read '^U'
read '^V'
read '^W'
read '^X'
read '^Y'
read '^Z'
read '^['
read '^\'
read '^]'
read '^^'
read '^_'
read ''
read '!'
read '"'
read '#'
read '$'
read '%'
read '&'
read '''
read '('
read ')'
read '*'
read '+'
read ','
read '-'
read '.'
read '/'
read '0'
read '1'
read '2'
read '3'
read '4'
read '5'
read '6'
read '7'
read '8'
read '9'
read ':'
read ';'
read '<'
read '='
read '>'
read '?'
read '@'
read 'A'
read 'B'
read 'C'
read 'D'
read 'E'
read 'F'
read 'G'
read 'H'
read 'I'
read 'J'
read 'K'
read 'L'
read 'M'
read 'N'
read 'O'
read 'P'
read 'Q'
read 'R'
read 'S'
read 'T'
read 'U'
read 'V'
read 'W'
read 'X'
read 'Y'
read 'Z'
read '['
read ']'
read '^'
read '_'
read '`'
read 'a'
read 'b'
read 'c'
read 'd'
read 'e'
read 'f'
read 'g'
read 'h'
read 'i'
read 'j'
read 'k'
read 'l'
read 'm'
read 'n'
read 'o'
read 'p'
read 'q'
read 'r'
read 's'
read 't'
read 'u'
read 'v'
read 'w'
read 'x'
read 'y'
read 'z'
read '{'
read '|'
read '}'
read '~'
read '^?'
read 'M-^@'
read 'M-^A'
read 'M-^B'
read 'M-^C'
read 'M-^D'
read 'M-^E'
read 'M-^F'
read 'M-^G'
read 'M-^H'
read 'M-^I'
read 'M-
'
read 'M-^K'
read 'M-^L'
read 'M-^M'
read 'M-^N'
read 'M-^O'
read 'M-^P'
read 'M-^Q'
read 'M-^R'
read 'M-^S'
read 'M-^T'
read 'M-^U'
read 'M-^V'
read 'M-^W'
read 'M-^X'
read 'M-^Y'
read 'M-^Z'
read 'M-^['
read 'M-^\'
read 'M-^]'
read 'M-^^'
read 'M-^_'
read 'M- '
read 'M-!'
read 'M-"'
read 'M-#'
read 'M-$'
read 'M-%'
read 'M-&'
read 'M-''
read 'M-('
read 'M-)'
read 'M-*'
read 'M-+'
read 'M-,'
read 'M--'
read 'M-.'
read 'M-/'
read 'M-0'
read 'M-1'
read 'M-2'
read 'M-3'
read 'M-4'
read 'M-5'
read 'M-6'
read 'M-7'
read 'M-8'
read 'M-9'
read 'M-:'
read 'M-;'
read 'M-<'
read 'M-='
read 'M->'
read 'M-?'
read 'M-@'
read 'M-A'
read 'M-B'
read 'M-C'
read 'M-D'
read 'M-E'
read 'M-F'
read 'M-G'
read 'M-H'
read 'M-I'
read 'M-J'
read 'M-K'
read 'M-L'
read 'M-M'
read 'M-N'
read 'M-O'
read 'M-P'
read 'M-Q'
read 'M-R'
read 'M-S'
read 'M-T'
read 'M-U'
read 'M-V'
read 'M-W'
read 'M-X'
read 'M-Y'
read 'M-Z'
read 'M-['
read 'M-\'
read 'M-]'
read 'M-^'
read 'M-_'
read 'M-`'
read 'M-a'
read 'M-b'
read 'M-c'
read 'M-d'
read 'M-e'
read 'M-f'
read 'M-g'
read 'M-h'
read 'M-i'
read 'M-j'
read 'M-k'
read 'M-l'
read 'M-m'
read 'M-n'
read 'M-o'
read 'M-p'
read 'M-q'
read 'M-r'
read 'M-s'
read 'M-t'
read 'M-u'
read 'M-v'
read 'M-w'
read 'M-x'
read 'M-y'
read 'M-z'
read 'M-{'
read 'M-|'
read 'M-}'
read 'M-~'
read 'M-^?'
$ all256|while read -en 1 c; do  echo read "'$c'"; done|cat -vt
read ''
read '^A'
read '^B'
read '^C'
read '^D'
read '^E'
read '^F'
read '^G'
read '^H'
read ''
read ''
read '^K'
read '^L'
read '^M'
read '^N'
read '^O'
read '^P'
read '^Q'
read '^R'
read '^S'
read '^T'
read '^U'
read '^V'
read '^W'
read '^X'
read '^Y'
read '^Z'
read '^['
read '^\'
read '^]'
read '^^'
read '^_'
read ''
read '!'
read '"'
read '#'
read '$'
read '%'
read '&'
read '''
read '('
read ')'
read '*'
read '+'
read ','
read '-'
read '.'
read '/'
read '0'
read '1'
read '2'
read '3'
read '4'
read '5'
read '6'
read '7'
read '8'
read '9'
read ':'
read ';'
read '<'
read '='
read '>'
read '?'
read '@'
read 'A'
read 'B'
read 'C'
read 'D'
read 'E'
read 'F'
read 'G'
read 'H'
read 'I'
read 'J'
read 'K'
read 'L'
read 'M'
read 'N'
read 'O'
read 'P'
read 'Q'
read 'R'
read 'S'
read 'T'
read 'U'
read 'V'
read 'W'
read 'X'
read 'Y'
read 'Z'
read '['
read ']'
read '^'
read '_'
read '`'
read 'a'
read 'b'
read 'c'
read 'd'
read 'e'
read 'f'
read 'g'
read 'h'
read 'i'
read 'j'
read 'k'
read 'l'
read 'm'
read 'n'
read 'o'
read 'p'
read 'q'
read 'r'
read 's'
read 't'
read 'u'
read 'v'
read 'w'
read 'x'
read 'y'
read 'z'
read '{'
read '|'
read '}'
read '~'
read '^?'
read 'M-^@'
read 'M-^A'
read 'M-^B'
read 'M-^C'
read 'M-^D'
read 'M-^E'
read 'M-^F'
read 'M-^G'
read 'M-^H'
read 'M-^I'
read 'M-
'
read 'M-^K'
read 'M-^L'
read 'M-^M'
read 'M-^N'
read 'M-^O'
read 'M-^P'
read 'M-^Q'
read 'M-^R'
read 'M-^S'
read 'M-^T'
read 'M-^U'
read 'M-^V'
read 'M-^W'
read 'M-^X'
read 'M-^Y'
read 'M-^Z'
read 'M-^['
read 'M-^\'
read 'M-^]'
read 'M-^^'
read 'M-^_'
read 'M- '
read 'M-!'
read 'M-"'
read 'M-#'
read 'M-$'
read 'M-%'
read 'M-&'
read 'M-''
read 'M-('
read 'M-)'
read 'M-*'
read 'M-+'
read 'M-,'
read 'M--'
read 'M-.'
read 'M-/'
read 'M-0'
read 'M-1'
read 'M-2'
read 'M-3'
read 'M-4'
read 'M-5'
read 'M-6'
read 'M-7'
read 'M-8'
read 'M-9'
read 'M-:'
read 'M-;'
read 'M-<'
read 'M-='
read 'M->'
read 'M-?'
read 'M-@'
read 'M-A'
read 'M-B'
read 'M-C'
read 'M-D'
read 'M-E'
read 'M-F'
read 'M-G'
read 'M-H'
read 'M-I'
read 'M-J'
read 'M-K'
read 'M-L'
read 'M-M'
read 'M-N'
read 'M-O'
read 'M-P'
read 'M-Q'
read 'M-R'
read 'M-S'
read 'M-T'
read 'M-U'
read 'M-V'
read 'M-W'
read 'M-X'
read 'M-Y'
read 'M-Z'
read 'M-['
read 'M-\'
read 'M-]'
read 'M-^'
read 'M-_'
read 'M-`'
read 'M-a'
read 'M-b'
read 'M-c'
read 'M-d'
read 'M-e'
read 'M-f'
read 'M-g'
read 'M-h'
read 'M-i'
read 'M-j'
read 'M-k'
read 'M-l'
read 'M-m'
read 'M-n'
read 'M-o'
read 'M-p'
read 'M-q'
read 'M-r'
read 'M-s'
read 'M-t'
read 'M-u'
read 'M-v'
read 'M-w'
read 'M-x'
read 'M-y'
read 'M-z'
read 'M-{'
read 'M-|'
read 'M-}'
read 'M-~'
read 'M-^?'
$

Looks like $IFS characters and null become the empty string ''. The -e does not seem to change things. EOF is still handled the same.

Hi DGPickett...

I like what I see...

Ooooh, I never thought about binary file reading using "read" as the prime mover as I can easily work around hexdump, (or od), for my binary file handling.

Let me delve deeper and find a back door. <wink>

If I am defeated I will let you know...

I must be allowed the two character "\0" for any binary/byte zero however...

This will be really interesting for me - thanks...

bakunin

1 Like

Hi bakunin...

Thanks a lot; if only I was that good...

Hi DGPickett...

This is the best I can do. Although there is a lot of superfluous code the important bits can be seen. This uses a default Macbook Pro system using bash...

#!/bin/bash --posix

ls -l /tmp/binary.file

text=""
newtext=""
count=0

# Active part...
while read
do
	text=$(xxd -p /tmp/binary.file | tr -d '\n')
done < /tmp/binary.file

echo "*****************************************************"
echo -n "$text"
echo ""
echo "*****************************************************"

# Conversion part...
for count in $( seq 0 2 2047 )
do
	if [ "${text:$count:2}" == "00" ]
	then
		newtext=$newtext'\\0'
	else
		newtext=$newtext'\x'${text:$count:2}
	fi
done

# Binary string part...
text=$(echo -e -n "$newtext")
echo "*****************************************************"
echo -n "$text" > /tmp/newbinary.file
echo -n "$text"
echo ""
echo "*****************************************************"
hexdump -C /tmp/newbinary.file

The results on screen...

Last login: Sat Aug 24 21:32:46 on ttys000
AMIGA:barrywalker~> ./readtest.sh
-rw-r--r--  1 barrywalker  wheel  1024 24 Aug 21:18 /tmp/binary.file
*****************************************************
e8c28d4dc08d47000a19c075d84174527600d71dd80100b4e8e87cff8300070f00c99c01ffeb3d030009b70f78e78d00000300ff0000000538c748a304070d3f050aeb00300a00cb031100eb00f7c0551c890000895be5c7405378ab0083c40022d85689f0cd480fb880c34cf855538d184948434845004908485d644131c3c0484d540f48ff757d007248cd4d8bff5de80105ffff892483c2b14106ff078b854c53430e0d74ff4841895e8905f2750f0048e54c4848000000e874eb89858b0f3800003c76f62274c68375ffc375e8000048005b006448dec406e80600745148ff05f80989f6019e060077f41185e98d29ebebefff000048d93d8dd7c34cc7c60d8d00fac015ff41df3500c33349217589065effce8900090048276335c7bf000641008cc748002489484141410089854841310500e8ffbfff22c0059aba8dff3500ef00eff8003eac8a00455c8948488575e822170001c1ff0f460289594875ff756306c400633d89030848858989808bff48ff1cbc0fff4548ffff740089f9c7e97541fff8008d41c63c00c6ff161048c048495e0440f600850f03fd00c08309640000538b01c60055455c4c4010894885a248aadb834800ff48ac484148df74ec00ff1fcc00224930ec004808e848f7ed04fd488d54004c494c894848c07400488df706fb05058d4914c889feff0004d5ff01004c45038b00754583670089ff627fff00084c8dff8de7e7d985e7ff7c00853e008b31ff55bec3fdc0850100050009e85d574900481389f205488385eb4b00ea004889030074e8000089e7008900c05b4803db0005480cc700ee41c700fbffebb28b0048065be70a034c00831b80ac838318c1c0ff008d418bfe3b44e848e883e88dffadb27dd0742674f8820b29ffec0089c68b4945745d76001d00d14800dd8b483063413f09ba0538001ec69500080085251f85090048ff8a19e8458b030005003de8e675544c253d785541cade8d412b0e097474fffd00e8003b00000a888d18ffc0fe8874040000fe8d85ffe9090fff83484c91c0b5de06c3c0080a1651458daf8983c100c0fbc08989007440ff242d7400c4484c48d8010805e08b45c889ed8d94c64841480400f300551806e85d01e9a40000207c46f6ff00005338608323f800c1ff028901c0be4c00481c4889404555e80007884c3405730808ca6344e90063f65745f6ebe882540125fffb785341f7241018ebe8e87e850000ff48a3485d9045413b3fff48fe45ecffbf0548ff45050a488dfd39531c97088b8688634874009cdfb8859c004844bf3dbe8d00858a057848e8bec9e84889af0848ff52484870ffbd021fd6770800c7040e002b00b50589fc8900fc024529ff8b0000fc0100ff4c8b0000ac04003d108b0575e8ff7989e8bd007d08ff8dfe1900418d00488d85080fff067485fb83
*****************************************************
*****************************************************
?M??G\0
?u?AtRv\0??\0???|??\0\0??=\0	?x?\0\0\0?\0\0\08?H?
?\00
\0?\0?\0??U?\0\0?[??@Sx?\0??\0"?V???H???L?US?IHCHE\0H]dA1??HMTH?u}\0rH?M??]????$t?HA?^??u\0H?LHH\0\0\0?t?8\0\0<v?"tu??u?\0\0H\0[\0dH???\0tQH??	???\0w???\0???A?5\0?3I!u?^?\0	\0H'c5\0A\0??H\0$?HAAA\0??HA1\0????"?????5\0?\0??\0>??\0E\?HH?u?"\0??F?YHu?uc?\0c=H??????H???EH??t\0????uA??\0?A?<\0??H?HI^@?\0??\0??	d\0\0S??\0UE\L@?H??H?H\0?H?HAH?t?\0??\0"I0?\0?H???H?T\0LIL?HH?t\0H????I??\0??\0LE?\0uE?g\0??b?\L???????|\0?>\0?1?U?????\0\0	?]WI\0H??H???K\0?\0H?\0t?\0\0??\0?\0?[H?\0H
           ?\0?A?\0??\0H[?
L\0???????\0?A??;D?H?????}?t&t??
                                )??\0?IEt]v\0\0?H\0H0cA?	?8\0\\0?%?	\0H???E?\0\0=??uTL%=xUA?A+	tt??\0?\0;\0\0
??????t\0\0?????	??HL??????
QE?????\0?????\0t@?$-t\0?HLH??E?HAH\0?\0U?]?\0\0 |F??\0\0S8`?#?\0?????L\0HH?@EU?\0?L?cD?\0c?WE???T%??xSA?$???~?\0\0?H?H]?EA;??H?E???H?E
H??9S???cHt\0???\0HD?=??\0??xH???H?H?RHHp???\0?\0+\0????\0?E)??\0\0?\0?L?\0\0?\0=?u??y??\0???\0A?\0H??t???
*****************************************************
00000000  e8 c2 8d 4d c0 8d 47 5c  30 0a 19 c0 75 d8 41 74  |...M..G\0...u.At|
00000010  52 76 5c 30 d7 1d d8 01  5c 30 b4 e8 e8 7c ff 83  |Rv\0....\0...|..|
00000020  5c 30 07 0f 5c 30 c9 9c  01 ff eb 3d 03 5c 30 09  |\0..\0.....=.\0.|
00000030  b7 0f 78 e7 8d 5c 30 5c  30 03 5c 30 ff 5c 30 5c  |..x..\0\0.\0.\0\|
00000040  30 5c 30 05 38 c7 48 a3  04 07 0d 3f 05 0a eb 5c  |0\0.8.H....?...\|
00000050  30 30 0a 5c 30 cb 03 11  5c 30 eb 5c 30 f7 c0 55  |00.\0...\0.\0..U|
00000060  1c 89 5c 30 5c 30 89 5b  e5 c7 40 53 78 ab 5c 30  |..\0\0.[..@Sx.\0|
00000070  83 c4 5c 30 22 d8 56 89  f0 cd 48 0f b8 80 c3 4c  |..\0".V...H....L|
00000080  f8 55 53 8d 18 49 48 43  48 45 5c 30 49 08 48 5d  |.US..IHCHE\0I.H]|
00000090  64 41 31 c3 c0 48 4d 54  0f 48 ff 75 7d 5c 30 72  |dA1..HMT.H.u}\0r|
000000a0  48 cd 4d 8b ff 5d e8 01  05 ff ff 89 24 83 c2 b1  |H.M..]......$...|
000000b0  41 06 ff 07 8b 85 4c 53  43 0e 0d 74 ff 48 41 89  |A.....LSC..t.HA.|
000000c0  5e 89 05 f2 75 0f 5c 30  48 e5 4c 48 48 5c 30 5c  |^...u.\0H.LHH\0\|
000000d0  30 5c 30 e8 74 eb 89 85  8b 0f 38 5c 30 5c 30 3c  |0\0.t.....8\0\0<|
000000e0  76 f6 22 74 c6 83 75 ff  c3 75 e8 5c 30 5c 30 48  |v."t..u..u.\0\0H|
000000f0  5c 30 5b 5c 30 64 48 de  c4 06 e8 06 5c 30 74 51  |\0[\0dH.....\0tQ|
00000100  48 ff 05 f8 09 89 f6 01  9e 06 5c 30 77 f4 11 85  |H.........\0w...|
00000110  e9 8d 29 eb eb ef ff 5c  30 5c 30 48 d9 3d 8d d7  |..)....\0\0H.=..|
00000120  c3 4c c7 c6 0d 8d 5c 30  fa c0 15 ff 41 df 35 5c  |.L....\0....A.5\|
00000130  30 c3 33 49 21 75 89 06  5e ff ce 89 5c 30 09 5c  |0.3I!u..^...\0.\|
00000140  30 48 27 63 35 c7 bf 5c  30 06 41 5c 30 8c c7 48  |0H'c5..\0.A\0..H|
00000150  5c 30 24 89 48 41 41 41  5c 30 89 85 48 41 31 05  |\0$.HAAA\0..HA1.|
00000160  5c 30 e8 ff bf ff 22 c0  05 9a ba 8d ff 35 5c 30  |\0...."......5\0|
00000170  ef 5c 30 ef f8 5c 30 3e  ac 8a 5c 30 45 5c 89 48  |.\0..\0>..\0E\.H|
00000180  48 85 75 e8 22 17 5c 30  01 c1 ff 0f 46 02 89 59  |H.u.".\0....F..Y|
00000190  48 75 ff 75 63 06 c4 5c  30 63 3d 89 03 08 48 85  |Hu.uc..\0c=...H.|
000001a0  89 89 80 8b ff 48 ff 1c  bc 0f ff 45 48 ff ff 74  |.....H.....EH..t|
000001b0  5c 30 89 f9 c7 e9 75 41  ff f8 5c 30 8d 41 c6 3c  |\0....uA..\0.A.<|
000001c0  5c 30 c6 ff 16 10 48 c0  48 49 5e 04 40 f6 5c 30  |\0....H.HI^.@.\0|
000001d0  85 0f 03 fd 5c 30 c0 83  09 64 5c 30 5c 30 53 8b  |....\0...d\0\0S.|
000001e0  01 c6 5c 30 55 45 5c 4c  40 10 89 48 85 a2 48 aa  |..\0UE\L@..H..H.|
000001f0  db 83 48 5c 30 ff 48 ac  48 41 48 df 74 ec 5c 30  |..H\0.H.HAH.t.\0|
00000200  ff 1f cc 5c 30 22 49 30  ec 5c 30 48 08 e8 48 f7  |...\0"I0.\0H..H.|
00000210  ed 04 fd 48 8d 54 5c 30  4c 49 4c 89 48 48 c0 74  |...H.T\0LIL.HH.t|
00000220  5c 30 48 8d f7 06 fb 05  05 8d 49 14 c8 89 fe ff  |\0H.......I.....|
00000230  5c 30 04 d5 ff 01 5c 30  4c 45 03 8b 5c 30 75 45  |\0....\0LE..\0uE|
00000240  83 67 5c 30 89 ff 62 7f  ff 5c 30 08 4c 8d ff 8d  |.g\0..b..\0.L...|
00000250  e7 e7 d9 85 e7 ff 7c 5c  30 85 3e 5c 30 8b 31 ff  |......|\0.>\0.1.|
00000260  55 be c3 fd c0 85 01 5c  30 05 5c 30 09 e8 5d 57  |U......\0.\0..]W|
00000270  49 5c 30 48 13 89 f2 05  48 83 85 eb 4b 5c 30 ea  |I\0H....H...K\0.|
00000280  5c 30 48 89 03 5c 30 74  e8 5c 30 5c 30 89 e7 5c  |\0H..\0t.\0\0..\|
00000290  30 89 5c 30 c0 5b 48 03  db 5c 30 05 48 0c c7 5c  |0.\0.[H..\0.H..\|
000002a0  30 ee 41 c7 5c 30 fb ff  eb b2 8b 5c 30 48 06 5b  |0.A.\0.....\0H.[|
000002b0  e7 0a 03 4c 5c 30 83 1b  80 ac 83 83 18 c1 c0 ff  |...L\0..........|
000002c0  5c 30 8d 41 8b fe 3b 44  e8 48 e8 83 e8 8d ff ad  |\0.A..;D.H......|
000002d0  b2 7d d0 74 26 74 f8 82  0b 29 ff ec 5c 30 89 c6  |.}.t&t...)..\0..|
000002e0  8b 49 45 74 5d 76 5c 30  1d 5c 30 d1 48 5c 30 dd  |.IEt]v\0.\0.H\0.|
000002f0  8b 48 30 63 41 3f 09 ba  05 38 5c 30 1e c6 95 5c  |.H0cA?...8\0...\|
00000300  30 08 5c 30 85 25 1f 85  09 5c 30 48 ff 8a 19 e8  |0.\0.%...\0H....|
00000310  45 8b 03 5c 30 05 5c 30  3d e8 e6 75 54 4c 25 3d  |E..\0.\0=..uTL%=|
00000320  78 55 41 ca de 8d 41 2b  0e 09 74 74 ff fd 5c 30  |xUA...A+..tt..\0|
00000330  e8 5c 30 3b 5c 30 5c 30  0a 88 8d 18 ff c0 fe 88  |.\0;\0\0........|
00000340  74 04 5c 30 5c 30 fe 8d  85 ff e9 09 0f ff 83 48  |t.\0\0.........H|
00000350  4c 91 c0 b5 de 06 c3 c0  08 0a 16 51 45 8d af 89  |L..........QE...|
00000360  83 c1 5c 30 c0 fb c0 89  89 5c 30 74 40 ff 24 2d  |..\0.....\0t@.$-|
00000370  74 5c 30 c4 48 4c 48 d8  01 08 05 e0 8b 45 c8 89  |t\0.HLH......E..|
00000380  ed 8d 94 c6 48 41 48 04  5c 30 f3 5c 30 55 18 06  |....HAH.\0.\0U..|
00000390  e8 5d 01 e9 a4 5c 30 5c  30 20 7c 46 f6 ff 5c 30  |.]...\0\0 |F..\0|
000003a0  5c 30 53 38 60 83 23 f8  5c 30 c1 ff 02 89 01 c0  |\0S8`.#.\0......|
000003b0  be 4c 5c 30 48 1c 48 89  40 45 55 e8 5c 30 07 88  |.L\0H.H.@EU.\0..|
000003c0  4c 34 05 73 08 08 ca 63  44 e9 5c 30 63 f6 57 45  |L4.s...cD.\0c.WE|
000003d0  f6 eb e8 82 54 01 25 ff  fb 78 53 41 f7 24 10 18  |....T.%..xSA.$..|
000003e0  eb e8 e8 7e 85 5c 30 5c  30 ff 48 a3 48 5d 90 45  |...~.\0\0.H.H].E|
000003f0  41 3b 3f ff 48 fe 45 ec  ff bf 05 48 ff 45 05 0a  |A;?.H.E....H.E..|
00000400  48 8d fd 39 53 1c 97 08  8b 86 88 63 48 74 5c 30  |H..9S......cHt\0|
00000410  9c df b8 85 9c 5c 30 48  44 bf 3d be 8d 5c 30 85  |.....\0HD.=..\0.|
00000420  8a 05 78 48 e8 be c9 e8  48 89 af 08 48 ff 52 48  |..xH....H...H.RH|
00000430  48 70 ff bd 02 1f d6 77  08 5c 30 c7 04 0e 5c 30  |Hp.....w.\0...\0|
00000440  2b 5c 30 b5 05 89 fc 89  5c 30 fc 02 45 29 ff 8b  |+\0.....\0..E)..|
00000450  5c 30 5c 30 fc 01 5c 30  ff 4c 8b 5c 30 5c 30 ac  |\0\0..\0.L.\0\0.|
00000460  04 5c 30 3d 10 8b 05 75  e8 ff 79 89 e8 bd 5c 30  |.\0=...u..y...\0|
00000470  7d 08 ff 8d fe 19 5c 30  41 8d 5c 30 48 8d 85 08  |}.....\0A.\0H...|
00000480  0f ff 06 74 85 fb 83                              |...t...|
00000487
AMIGA:barrywalker~> 
    e8c28d4dc08d47000a19c075d84174
became
    00000000  e8 c2 8d 4d c0 8d 47 5c  30 0a 19 c0 75 d8 41 74  |...M..G\0...u.At|

Did you want to make the null more text friendly?

It looks like xxd was doing the lifting, not bash, with no reference to $REPLY (since you did 'while read' with no variable). Can you read, recognize and write a null in bash alone?

Hi DGPickett...

Yes and no. It is the only way I can get byte value zero into a string variable using only this default OSX install, as bash is not byte value zero friendly...

As for the xxd command, try and remove the file pointer after "done" and see what happens.
Try replacing it with say /dev/null too. I have absolutely no idea why /tmp/binary.file IS needed, but it is...

If you enter:-

echo -e -n "$text" > /tmp/somefilename

It will end up the same as /tmp/binary.file...

I have just had another idea.
Let me experiment and see if this idea works...

I will explain after some experimention what the idea is/was, whether successful or not, after I have tried it...

There is a wealth of possibilities, between the readline options, stty raw, and IFS, but so many tools do not deal with null, it's hard to even test. If you bash "read =n 1" a null into a variable, is it hard to tell the variable is the empty string '' or ""?

To write it out, one might use the printf character option, treating it as a special case and providing the null as a zero char (int) value. Once you get to treating it as a special case, you can write it with something like "echo '\0\c' ".

$IFS characters are another challenge, but if you use "read -n 1" you could do your own $IFS processing if any, setting $IFS to '' or unsetting it.

Getting byte zero, in pseudo_zero mode, into the shell is easy as the Arduino Voltmeter DEMO below shows.

However I have attempted to put byte zero into a string variable as a single character and I now concede defeat. It is not possible with default commands to force value of byte zero into any variable, so a minimum of "\0", 2 bytes, is required to _emulate_ it...

#!/bin/bash --posix
# $VER: Arduino.sh_Version_0.00.10_(C)2013_B.Walker_G0LCU.

Arduino_Device="Unknown!"
DATA="0"

> /tmp/usbdata.raw

echo "Remove the Arduino board from the USB port ,IF, it is connected."
read -p "Press <CR> to continue:- " -e Arduino_Device
echo "The error report IS expected!"
ls /dev/cu.usb*
echo "Now connect the Arduino board to the USB port."
sleep 5
read -p "Press <CR> to continue:- " -e Arduino_Device

Arduino_Device="`ls /dev/cu.usb*`"
clear
while true
do
	if [ ${#DATA} -le 0 ]
	then
		echo "ERROR! Unsuccessful data aquisition..."
		break
	fi
	DATA=$[ ( $DATA * 20 ) ]
	printf "\\x1B[11;33f0.000 Volts DC.\n\n"
	if [ ${#DATA} -eq 2 ]
	then
		printf "\\x1B[11;36f$DATA\n\n"
	fi
	if [ ${#DATA} -eq 3 ]
	then
		printf "\\x1B[11;35f$DATA\n\n"
	fi
	if [ ${#DATA} -eq 4 ]
	then
		DATA="${DATA:0:1}.${DATA:1:2}"
		printf "\\x1B[11;33f$DATA\n\n"
	fi
	dd if="$Arduino_Device" of=/tmp/usbdata.raw bs=1 count=1
	DATA=`hexdump -n1 -s0 -v -e '1/1 "%u"' /tmp/usbdata.raw`
done

Display like this:-

Last login: Fri Aug 30 21:34:09 on ttys000
AMIGA:barrywalker~> ./Arduino.sh
Remove the Arduino board from the USB port ,IF, it is connected.
Press <CR> to continue:- 
The error report IS expected!
ls: /dev/cu.usb*: No such file or directory
Now connect the Arduino board to the USB port.
Press <CR> to continue:- 

                                0.000 Volts DC.

1+0 records in
1+0 records out
1 bytes transferred in 0.007705 secs (130 bytes/sec)c)

# AND...

                                2.580 Volts DC.

1+0 records in
1+0 records out
1 bytes transferred in 0.007620 secs (131 bytes/sec)))

# AND...

                                5.100 Volts DC.

1+0 records in
1+0 records out
1 bytes transferred in 0.007678 secs (130 bytes/sec)))

Wel, in a null terminated string application, null is always '' the empty string, as it reterminates the string, just as a strcpy of a 5 byte plus null string into a 20 byte buffer initialized with 19 spaces and a null creates a string length of 5.

Now, is it the only character that is equal to '' the empty string? Does read differentiate that from EOF in a while read loop?

$ bash -c 'export ct=0 ; unset IFS ; all256|while read -rn 1 c
do
 echo "$(( ct++ )): '"'"'$c'"'"'"
done|cat -vt'
0: ''
1: '^A'
2: '^B'
3: '^C'
4: '^D'
5: '^E'
6: '^F'
7: '^G'
8: '^H'
9: ''
10: ''
11: '^K'
12: '^L'
13: '^M'
14: '^N'
15: '^O'
16: '^P'
17: '^Q'
18: '^R'
19: '^S'
20: '^T'
21: '^U'
22: '^V'
23: '^W'
24: '^X'
25: '^Y'
26: '^Z'
27: '^['
28: '^\'
29: '^]'
30: '^^'
31: '^_'
32: ''
33: '!'
34: '"'
35: '#'
36: '$'
37: '%'
38: '&'
39: '''
40: '('
41: ')'
42: '*'
43: '+'
44: ','
45: '-'
46: '.'
47: '/'
48: '0'
49: '1'
50: '2'
51: '3'
52: '4'
53: '5'
54: '6'
55: '7'
56: '8'
57: '9'
58: ':'
59: ';'
60: '<'
61: '='
62: '>'
63: '?'
64: '@'
65: 'A'
66: 'B'
67: 'C'
68: 'D'
69: 'E'
70: 'F'
71: 'G'
72: 'H'
73: 'I'
74: 'J'
75: 'K'
76: 'L'
77: 'M'
78: 'N'
79: 'O'
80: 'P'
81: 'Q'
82: 'R'
83: 'S'
84: 'T'
85: 'U'
86: 'V'
87: 'W'
88: 'X'
89: 'Y'
90: 'Z'
91: '['
92: '\'
93: ']'
94: '^'
95: '_'
96: '`'
97: 'a'
98: 'b'
99: 'c'
100: 'd'
101: 'e'
102: 'f'
103: 'g'
104: 'h'
105: 'i'
106: 'j'
107: 'k'
108: 'l'
109: 'm'
110: 'n'
111: 'o'
112: 'p'
113: 'q'
114: 'r'
115: 's'
116: 't'
117: 'u'
118: 'v'
119: 'w'
120: 'x'
121: 'y'
122: 'z'
123: '{'
124: '|'
125: '}'
126: '~'
127: '^?'
128: 'M-^@'
129: 'M-^A'
130: 'M-^B'
131: 'M-^C'
132: 'M-^D'
133: 'M-^E'
134: 'M-^F'
135: 'M-^G'
136: 'M-^H'
137: 'M-^I'
138: 'M-
'
139: 'M-^K'
140: 'M-^L'
141: 'M-^M'
142: 'M-^N'
143: 'M-^O'
144: 'M-^P'
145: 'M-^Q'
146: 'M-^R'
147: 'M-^S'
148: 'M-^T'
149: 'M-^U'
150: 'M-^V'
151: 'M-^W'
152: 'M-^X'
153: 'M-^Y'
154: 'M-^Z'
155: 'M-^['
156: 'M-^\'
157: 'M-^]'
158: 'M-^^'
159: 'M-^_'
160: 'M- '
161: 'M-!'
162: 'M-"'
163: 'M-#'
164: 'M-$'
165: 'M-%'
166: 'M-&'
167: 'M-''
168: 'M-('
169: 'M-)'
170: 'M-*'
171: 'M-+'
172: 'M-,'
173: 'M--'
174: 'M-.'
175: 'M-/'
176: 'M-0'
177: 'M-1'
178: 'M-2'
179: 'M-3'
180: 'M-4'
181: 'M-5'
182: 'M-6'
183: 'M-7'
184: 'M-8'
185: 'M-9'
186: 'M-:'
187: 'M-;'
188: 'M-<'
189: 'M-='
190: 'M->'
191: 'M-?'
192: 'M-@'
193: 'M-A'
194: 'M-B'
195: 'M-C'
196: 'M-D'
197: 'M-E'
198: 'M-F'
199: 'M-G'
200: 'M-H'
201: 'M-I'
202: 'M-J'
203: 'M-K'
204: 'M-L'
205: 'M-M'
206: 'M-N'
207: 'M-O'
208: 'M-P'
209: 'M-Q'
210: 'M-R'
211: 'M-S'
212: 'M-T'
213: 'M-U'
214: 'M-V'
215: 'M-W'
216: 'M-X'
217: 'M-Y'
218: 'M-Z'
219: 'M-['
220: 'M-\'
221: 'M-]'
222: 'M-^'
223: 'M-_'
224: 'M-`'
225: 'M-a'
226: 'M-b'
227: 'M-c'
228: 'M-d'
229: 'M-e'
230: 'M-f'
231: 'M-g'
232: 'M-h'
233: 'M-i'
234: 'M-j'
235: 'M-k'
236: 'M-l'
237: 'M-m'
238: 'M-n'
239: 'M-o'
240: 'M-p'
241: 'M-q'
242: 'M-r'
243: 'M-s'
244: 'M-t'
245: 'M-u'
246: 'M-v'
247: 'M-w'
248: 'M-x'
249: 'M-y'
250: 'M-z'
251: 'M-{'
252: 'M-|'
253: 'M-}'
254: 'M-~'
255: 'M-^?'
$

Too many aliases. Whatever is or defaults to IFS is indistinguishable from null:

$ bash -c 'export ct=0 IFS="
" ; all256|while read -ern 1 c
do
 echo "$(( ct++ )): '"'"'$c'"'"'"
done|cat -vt'
0: ''
1: '^A'
2: '^B'
3: '^C'
4: '^D'
5: '^E'
6: '^F'
7: '^G'
8: '^H'
9: '^I'
10: ''
11: '^K'
12: '^L'
13: '^M'
14: '^N'
15: '^O'
16: '^P'
17: '^Q'
18: '^R'
19: '^S'
20: '^T'
21: '^U'
22: '^V'
23: '^W'
24: '^X'
25: '^Y'
26: '^Z'
27: '^['
28: '^\'
29: '^]'
30: '^^'
31: '^_'
32: ' '
33: '!'
34: '"'
35: '#'
36: '$'
37: '%'
38: '&'
39: '''
40: '('
41: ')'
42: '*'
43: '+'
44: ','
45: '-'
46: '.'
47: '/'
48: '0'
49: '1'
50: '2'
51: '3'
52: '4'
53: '5'
54: '6'
55: '7'
56: '8'
57: '9'
58: ':'
59: ';'
60: '<'
61: '='
62: '>'
63: '?'
64: '@'
65: 'A'
66: 'B'
67: 'C'
68: 'D'
69: 'E'
70: 'F'
71: 'G'
72: 'H'
73: 'I'
74: 'J'
75: 'K'
76: 'L'
77: 'M'
78: 'N'
79: 'O'
80: 'P'
81: 'Q'
82: 'R'
83: 'S'
84: 'T'
85: 'U'
86: 'V'
87: 'W'
88: 'X'
89: 'Y'
90: 'Z'
91: '['
92: '\'
93: ']'
94: '^'
95: '_'
96: '`'
97: 'a'
98: 'b'
99: 'c'
100: 'd'
101: 'e'
102: 'f'
103: 'g'
104: 'h'
105: 'i'
106: 'j'
107: 'k'
108: 'l'
109: 'm'
110: 'n'
111: 'o'
112: 'p'
113: 'q'
114: 'r'
115: 's'
116: 't'
117: 'u'
118: 'v'
119: 'w'
120: 'x'
121: 'y'
122: 'z'
123: '{'
124: '|'
125: '}'
126: '~'
127: '^?'
128: 'M-^@'
129: 'M-^A'
130: 'M-^B'
131: 'M-^C'
132: 'M-^D'
133: 'M-^E'
134: 'M-^F'
135: 'M-^G'
136: 'M-^H'
137: 'M-^I'
138: 'M-
'
139: 'M-^K'
140: 'M-^L'
141: 'M-^M'
142: 'M-^N'
143: 'M-^O'
144: 'M-^P'
145: 'M-^Q'
146: 'M-^R'
147: 'M-^S'
148: 'M-^T'
149: 'M-^U'
150: 'M-^V'
151: 'M-^W'
152: 'M-^X'
153: 'M-^Y'
154: 'M-^Z'
155: 'M-^['
156: 'M-^\'
157: 'M-^]'
158: 'M-^^'
159: 'M-^_'
160: 'M- '
161: 'M-!'
162: 'M-"'
163: 'M-#'
164: 'M-$'
165: 'M-%'
166: 'M-&'
167: 'M-''
168: 'M-('
169: 'M-)'
170: 'M-*'
171: 'M-+'
172: 'M-,'
173: 'M--'
174: 'M-.'
175: 'M-/'
176: 'M-0'
177: 'M-1'
178: 'M-2'
179: 'M-3'
180: 'M-4'
181: 'M-5'
182: 'M-6'
183: 'M-7'
184: 'M-8'
185: 'M-9'
186: 'M-:'
187: 'M-;'
188: 'M-<'
189: 'M-='
190: 'M->'
191: 'M-?'
192: 'M-@'
193: 'M-A'
194: 'M-B'
195: 'M-C'
196: 'M-D'
197: 'M-E'
198: 'M-F'
199: 'M-G'
200: 'M-H'
201: 'M-I'
202: 'M-J'
203: 'M-K'
204: 'M-L'
205: 'M-M'
206: 'M-N'
207: 'M-O'
208: 'M-P'
209: 'M-Q'
210: 'M-R'
211: 'M-S'
212: 'M-T'
213: 'M-U'
214: 'M-V'
215: 'M-W'
216: 'M-X'
217: 'M-Y'
218: 'M-Z'
219: 'M-['
220: 'M-\'
221: 'M-]'
222: 'M-^'
223: 'M-_'
224: 'M-`'
225: 'M-a'
226: 'M-b'
227: 'M-c'
228: 'M-d'
229: 'M-e'
230: 'M-f'
231: 'M-g'
232: 'M-h'
233: 'M-i'
234: 'M-j'
235: 'M-k'
236: 'M-l'
237: 'M-m'
238: 'M-n'
239: 'M-o'
240: 'M-p'
241: 'M-q'
242: 'M-r'
243: 'M-s'
244: 'M-t'
245: 'M-u'
246: 'M-v'
247: 'M-w'
248: 'M-x'
249: 'M-y'
250: 'M-z'
251: 'M-{'
252: 'M-|'
253: 'M-}'
254: 'M-~'
255: 'M-^?'
$

You have whetted my apetite again... ;o)

There is possibly one _bug_ I can see in both my and your attempts is character 92, "\"
causing a pseudo escape on the next binary character it sees when running an
"somevar=echo -e......" into it more than once.

It also seems your 0x09 and 0x0A become NULL too.

I can understand why 0x0A _might_ occur but not 0x09, (Tab).

On a binary file with single byte zero(s) uisng the default bash "read", yes, but on a
binary_string_variable, hmmm, hard to judge as placing byte zero into a variable is
not possible at this point.

Ignoring "read" FTTB I am going to try and force byte zero into the middlie of
a "Hello World." string, i.e. the space replaced with 0x00...

---------- Post updated 07-09-13 at 11:21 AM ---------- Previous update was 06-09-13 at 10:58 PM ----------

Hi DGPickett...
I tried this out from the command line and......

Last login: Sat Sep  7 10:17:43 on ttys000
AMIGA:barrywalker~> text="Hello World."
AMIGA:barrywalker~> text=${text/ /"\0"}
AMIGA:barrywalker~> echo ${#text}
13
AMIGA:barrywalker~> echo -n $text > /tmp/txt
AMIGA:barrywalker~> hexdump -C /tmp/txt
00000000  48 65 6c 6c 6f 5c 30 57  6f 72 6c 64 2e           |Hello\0World.|
0000000d
AMIGA:barrywalker~> echo -e -n $text > /tmp/txt
AMIGA:barrywalker~> hexdump -C /tmp/txt
00000000  48 65 6c 6c 6f 00 57 6f  72 6c 64 2e              |Hello.World.|
0000000c
AMIGA:barrywalker~> echo -e "$text"
HelloWorld.
AMIGA:barrywalker~> echo "$text"
Hello\0World.
AMIGA:barrywalker~> read newtext < /tmp/txt
AMIGA:barrywalker~> echo $newtext
Hello
AMIGA:barrywalker~> echo ${#newtext}
5
AMIGA:barrywalker~> # 0x00 always strips anything after it...
AMIGA:barrywalker~>
AMIGA:barrywalker~> echo -n $text > /tmp/txt
AMIGA:barrywalker~> read newtext < /tmp/txt
AMIGA:barrywalker~> echo $newtext
Hello0World.
AMIGA:barrywalker~> echo ${#newtext}
12
AMIGA:barrywalker~> hexdump -C /tmp/txt
00000000  48 65 6c 6c 6f 5c 30 57  6f 72 6c 64 2e           |Hello\0World.|
0000000d
AMIGA:barrywalker~> _

"read" has brought in the correct length of "Hello World.", 12, with the space shown as
character 0, but its file length is actually 13 characters in size...

So to return back to 13 characters again using pseudo-zero 2 backslashes will be needed
inside the file now making the file length 14.

This is a real killer for binary_string to file and back again manipulation.
I think the only way to account for binary zero is to use "\0" two characters and detect
those two characters inside the binary_string something like this...

if [ "${binary_string:$some_position:2}" == "\0" ]
then
        echo "\0 at position $some_position in binary string."
        # Do other binary string manipulation as required.
fi

The problem is that there may be two genuine independent "\" and "0" characters side
by side and not related to binary zero at all, just independent characters...

This is getting more difficult by the day... ;o)

So, you can see shell is not a binary capable tool. Some bytes it can handle, but nulls, linefeeds, white space are problematic. But you can write trivial C to support it, like at tool that just does a tr of null to something else, trnull:

 
#include <stdio.h>
 
int main( int argc, char **argv ){
  int c ;
  if ( argc != 2 || strlen( argv[1] ) != 1 ){
    fputs(
"\n"
"Usage: trnull <byte_for_null>\n"
"\n"
"Translates null to <byte_for_null>, reading stdin and writing stdout.\n"
"\n", stderr );
    exit( 1 );
   }
  while ( EOF != ( c = getchar() )){
    if ( EOF == putchar( c ? c : argv[1][0] )){
      if ( ferror( stdout )){
        perror( "stdout" );
        exit( 2 );
       }
     }
   }
  if ( ferror( stdin )){
    perror( "stdin" );
    exit( 3 );
   }
 }
"mysrc/trnull.c" 31 lines, 541 characters 
$ mycc trnull
$ trnull
 
Usage: trnull <byte_for_null>
 
Translates null to <byte_for_null>, reading stdin and writing stdout.
 
$ all256|trnull '!'|od -bc
0000000   ! 001 002 003 004 005 006 007  \b  \t  \n 013  \f  \r 016 017
        041 001 002 003 004 005 006 007 010 011 012 013 014 015 016 017
0000020 020 021 022 023 024 025 026 027 030 031 032 033 034 035 036 037
        020 021 022 023 024 025 026 027 030 031 032 033 034 035 036 037
0000040       !   "   #   $   %   &   '   (   )   *   +   ,   -   .   /
        040 041 042 043 044 045 046 047 050 051 052 053 054 055 056 057
0000060   0   1   2   3   4   5   6   7   8   9   :   ;   <   =   >   ?
        060 061 062 063 064 065 066 067 070 071 072 073 074 075 076 077
0000100   @   A   B   C   D   E   F   G   H   I   J   K   L   M   N   O
        100 101 102 103 104 105 106 107 110 111 112 113 114 115 116 117
0000120   P   Q   R   S   T   U   V   W   X   Y   Z   [   \   ]   ^   _
        120 121 122 123 124 125 126 127 130 131 132 133 134 135 136 137
0000140   `   a   b   c   d   e   f   g   h   i   j   k   l   m   n   o
        140 141 142 143 144 145 146 147 150 151 152 153 154 155 156 157
0000160   p   q   r   s   t   u   v   w   x   y   z   {   |   }   ~ 177
        160 161 162 163 164 165 166 167 170 171 172 173 174 175 176 177
0000200 200 201 202 203 204 205 206 207 210 211 212 213 214 215 216 217
        200 201 202 203 204 205 206 207 210 211 212 213 214 215 216 217
0000220 220 221 222 223 224 225 226 227 230 231 232 233 234 235 236 237
        220 221 222 223 224 225 226 227 230 231 232 233 234 235 236 237
0000240 240 241 242 243 244 245 246 247 250 251 252 253 254 255 256 257
        240 241 242 243 244 245 246 247 250 251 252 253 254 255 256 257
0000260 260 261 262 263 264 265 266 267 270 271 272 273 274 275 276 277
        260 261 262 263 264 265 266 267 270 271 272 273 274 275 276 277
0000300 300 301 302 303 304 305 306 307 310 311 312 313 314 315 316 317
        300 301 302 303 304 305 306 307 310 311 312 313 314 315 316 317
0000320 320 321 322 323 324 325 326 327 330 331 332 333 334 335 336 337
        320 321 322 323 324 325 326 327 330 331 332 333 334 335 336 337
0000340 340 341 342 343 344 345 346 347 350 351 352 353 354 355 356 357
        340 341 342 343 344 345 346 347 350 351 352 353 354 355 356 357
0000360 360 361 362 363 364 365 366 367 370 371 372 373 374 375 376 377
        360 361 362 363 364 365 366 367 370 371 372 373 374 375 376 377
0000400
$

Hi DGP...

0000000   ! 001 002 003 004 005 006 007  \b  \t  \n 013  \f  \r 016 017
        041 001 002 003 004 005 006 007 010 011 012 013 014 015 016 017
0000020 020 021 022 023 024 025 026 027 030 031 032 033 034 035 036 037
        020 021 022 023 024 025 026 027 030 031 032 033 034 035 036 037
0000040       !   "   #   $   %   &   '   (   )   *   +   ,   -   .   /
        040 041 042 043 044 045 046 047 050 051 052 053 054 055 056 057
0000060   0   1   2   3   4   5   6   7   8   9   :   ;   <   =   >   ?
        060 061 062 063 064 065 066 067 070 071 072 073 074 075 076 077

The problem is the same as mine, in that address 0x41 is also a(n) "!" so is this a binary zero or a real "!".
There is no way of knowing so I thought of using "@@" to substitute for binary zero as "\@" would present a similar problem as my "\0", but, would import into "read" easily.
But again "@@" may actually exist as real characters and not a substitution but at least it is starter approach to emulating binary zero...

I'll give it a whirl tomorrow...

HTH...

All the 256 holes are always taken, so if your milieu does not have a limited character set, then you are right, the only escape is into longer strings for short characters, but then only context tells you that is the sort of file you have. That's why JAVA and xml use UTF-8 and label the content type. It started as a 'code page' discussion, a.k.a "what drum is on the printer".

There are apps for UNIX around that let you deal with image files like jpg and gif in text files of text numbers, like a decimal dump of a bmp file. I suppose if you run binary through the right dump you can then process it in shell back to a binary file.

Hi DGP...

Here is an idea using extended characters for 0x00, 0x0A and 0x1B that can easily be imported by the "read" command...

#!/bin/bash --posix
# Using OSX 10.7.5, bash default terminal...
# Alt-8 = � - zero...
bin_zero="�"
# Alt-7 = � - newline...
new_line="�"
# Alt-6 = � - escape...
esc_char="�"
# A simple string using �, �, and � as a real binary substitutes.
#
echo $bin_zero"This is a binary "$bin_zero$new_line$esc_char" zero, newline and escape character cop-out."$esc_char$bin_zero$new_line > /tmp/bin.dat
# Check it and echo has included a newline character for added fun...
hexdump -C /tmp/bin.dat
#
read bin_text < /tmp/bin.dat
echo "$bin_text"
# Newline is stripped as expected...
#
# Now sawp extended characters into escape characters.
text=$(echo "$bin_text" | sed 's/�/\\0/g;s/�/\\n/g;s/�/\\\\/g')
# Done!
# Put escape charatcer version of string into a file.
echo -n "$text" > /tmp/newbin.dat
# Done!
# Check that escape charaters for _newline_ and zero exist.
hexdump -C /tmp/newbin.dat
echo "$text"
# Done
# Now resave to the real file with binary 0x00s and 0x0A.
echo -e -n "$text" > /tmp/newbin.dat
# Done
# Prove it exists.
hexdump -C /tmp/newbin.dat
echo "Now converted to binary using extended characters..."
# Yup, it does...

The results:-

Now converted to binary using extended characters...
AMIGA:barrywalker~> ./bin_test.sh
00000000  e2 80 a2 54 68 69 73 20  69 73 20 61 20 62 69 6e  |...This is a bin|
00000010  61 72 79 20 e2 80 a2 c2  b6 c2 a7 20 7a 65 72 6f  |ary ....... zero|
00000020  2c 20 6e 65 77 6c 69 6e  65 20 61 6e 64 20 65 73  |, newline and es|
00000030  63 61 70 65 20 63 68 61  72 61 63 74 65 72 20 63  |cape character c|
00000040  6f 70 2d 6f 75 74 2e c2  a7 e2 80 a2 c2 b6 0a     |op-out.........|
0000004f
�This is a binary ��� zero, newline and escape character cop-out.���
00000000  5c 30 54 68 69 73 20 69  73 20 61 20 62 69 6e 61  |\0This is a bina|
00000010  72 79 20 5c 30 5c 6e 5c  5c 20 7a 65 72 6f 2c 20  |ry \0\n\\ zero, |
00000020  6e 65 77 6c 69 6e 65 20  61 6e 64 20 65 73 63 61  |newline and esca|
00000030  70 65 20 63 68 61 72 61  63 74 65 72 20 63 6f 70  |pe character cop|
00000040  2d 6f 75 74 2e 5c 5c 5c  30 5c 6e                 |-out.\\\0\n|
0000004b
\0This is a binary \0\n\\ zero, newline and escape character cop-out.\\\0\n
00000000  00 54 68 69 73 20 69 73  20 61 20 62 69 6e 61 72  |.This is a binar|
00000010  79 20 00 0a 5c 20 7a 65  72 6f 2c 20 6e 65 77 6c  |y ..\ zero, newl|
00000020  69 6e 65 20 61 6e 64 20  65 73 63 61 70 65 20 63  |ine and escape c|
00000030  68 61 72 61 63 74 65 72  20 63 6f 70 2d 6f 75 74  |haracter cop-out|
00000040  2e 5c 00 0a                                       |.\..|
00000044
Now converted to binary using extended characters...
AMIGA:barrywalker~> 

EDIT:

I didn't want to use "sed" but longhand became a large program and defeated the object...

Noticed a minor bug and squashed it...