Conversion of Binary csv file

UNIX for Beginners Q & A
1

Hello,
I have a binary csv file that was created on 'Red Hat Enterprise Linux Server release 6.6'. Now we have transferred all files on Ubuntu 16.04.2 LTS/xenial
On opening the file in Ubuntu, there are special characters

<FF><FE>1^@,^@S^@1^@_^@5^@0^@2^@0^@4^@,^@M^@u^@s^@_^@C^@h^@,^@T^@A^@A^@G^@G^@C^@G^@A^@-^@G^@C^@G^@T^@A^@A^@G^@A^@
^@2^@,^@S^@2^@_^@7^@6^@7^@7^@5^@,^@M^@u^@s^@_^@C^@h^@,^@C^@G^@T^@A^@C^@T^@A^@G^@-^@G^@C^@G^@T^@A^@A^@G^@A^@
^@3^@,^@S^@3^@_^@7^@9^@1^@3^@5^@,^@B^@r^@_^@C^@h^@,^@A^@G^@G^@C^@A^@G^@A^@A^@-^@G^@C^@G^@T^@A^@A^@G^@A^@
^@4^@,^@S^@4^@_^@7^@9^@1^@4^@8^@,^@B^@r^@_^@C^@h^@,^@T^@C^@C^@T^@G^@A^@G^@C^@-^@G^@C^@G^@T^@A^@A^@G^@A^@
^@5^@,^@S^@5^@_^@7^@9^@8^@4^@3^@,^@B^@r^@_^@C^@h^@,^@G^@G^@A^@C^@T^@C^@C^@T^@-^@G^@C^@G^@T^@A^@A^@G^@A^@

is there a way to remove the characters and keep the file as a binary file ? I tried the following that technically works but this does not keep the file binary anymore

iconv -f UTF-16LE -t UTF-8 config.csv > conv_config.csv
2

I don't think "binary" csv- files exist. The csv format was created to exchange data between systems in text form, fields separated by comma (or semicolon or ...). Trying iconv was a brilliant idea! So - use as is after conversion.

3

When i try to open the file in ReadHat, this is what it says

NBN@myComp:/mnt/NB/pipeline$less -S config.csv
"config.csv" may be a binary file.  See it anyway?

So I can continue to use iconv ?

4

Well, does it work? If yes, yes. If no, no.

5

Do you mean:-

if [ it-works ]
then
   continue-to-use-iconv
else
   try-another-way
fi

:stuck_out_tongue:

I suppose the real question is why you need it as a "binary file"? (not simple ASCII display text)

If you want to extract the content for some purpose, then any way to get at it that works is valid. If you want to update the file, then that's another question.

What do you actually want to achieve? If you can give us an idea of your thoughts then there may be an alternate way. It might help to say why you need a "binary file" in the first place.

Kind regards,
Robin

6

I don't believe s/he wants / needs a binary file. It seems more to be a discrepancy in (default?) coding between the two nodes.

7

Maybe the Red Hat file was accidentally encrypted. Easily done if one exits from vi using X instead of x. Post 3 indicates that the file is not text on the source machine.

8

I don't understand what you mean by a binary file here as ALL files are binary, so......
......Ignoring the <FF><FE> at the start for the time being is the result below what you are looking for, OR, do you want each character pair '^@' replacing with a single NULL character and '<FF><FE>' replacing with their single binary characters for 0xFF and 0xFE?

Last login: Thu Jul 13 20:46:07 on console
AMIGA:amiga~> cd Desktop/Code/Shell
AMIGA:amiga~/Desktop/Code/Shell> ./CTRL_AT.sh
<FF><FE>1,S1_50204,Mus_Ch,TAAGGCGA-GCGTAAGA
2,S2_76775,Mus_Ch,CGTACTAG-GCGTAAGA
3,S3_79135,Br_Ch,AGGCAGAA-GCGTAAGA
4,S4_79148,Br_Ch,TCCTGAGC-GCGTAAGA
5,S5_79843,Br_Ch,GGACTCCT-GCGTAAGA
AMIGA:amiga~/Desktop/Code/Shell> _
9

The <FF><FE> at the start is a BOM (Byte Order Mark), saying that the file is an UTF16-Little-Endian text file.
To "keep the file binary", which I read as "keep the encoding as it is", simply pipe the output of iconv to your prefered pager or whatever tool needs to process the file:

iconv -f UTF-16LE -t UTF-8 config.csv |less