Removing inline binary data from txt file

Shell Programming and Scripting
1

I am trying to parse a file but the filehas binary data inline mixed with text fields.
I tried the binutils strings function , it get the binary data out but put the char following the binary data in a new line .

input file

app_id:1936 pgm_num:0 branch:TBNY ord_num:0500012(�QMK) deal_num:0 ref_num:R
app_id:1396 pgm_num:0 branch:TBNY ord_num:0000016(�11S) deal_num:0 ref_num:R
app_id:1996 pgm_num:0 branch:TBNY ord_num:0260012(�PAK) deal_num:0 ref_num:R
app_id:1936 pgm_num:0 branch:TBNY ord_num:F016 deal_number:0 ref_num:R

I need to get to

app_id:1936 pgm_num:0 branch:TBNY ord_num:0500012() deal_num:0 ref_num:R
app_id:1396 pgm_num:0 branch:TBNY ord_num:0000016() deal_num:0 ref_num:R
app_id:1996 pgm_num:0 branch:TBNY ord_num:0260012() deal_num:0 ref_num:R
app_id:1936 pgm_num:0 branch:TBNY ord_num:F016 deal_number:0 ref_num:R

Just want to remove the binary data , the brackets surrounding the binary data are not material

any suggestions

This is breaking what I am trying to do here
My earlier q on parsing :

2

try:

perl -pe 's/[^[:ascii:]]//g' infile
3

There is some other binary data which is still causing this to file.
When copying to this editor it causes a line break

app_id:1996 pgm_num:0 branch:TBNY ord_num:0440010(
Q6I) deal_number:0

---------- Post updated at 12:11 PM ---------- Previous update was at 11:33 AM ----------

These are probabaly control chars ( not the ususal ^M ) ones .
Is there a way to remove all control characters from a file

tr ?

4

Try this gawk code:

gawk '{gsub(/[[:cntrl:]]/,x)}1' inputfile > outputfile
5

nope

sandbox]$>nawk '{gsub(/[[:cntrl:]]/,x)}1' bin4lines.txt
app_id:1936 pgm_num:0 branch:TBNY ord_num:0500012(QMK) deal_num:0 ref_num:R
app_id:1396 pgm_num:0 branch:TBNY ord_num:0000016(�11S) deal_num:0 ref_num:R
app_id:1996 pgm_num:0 branch:TBNY ord_num:0260012(�PAK) deal_num:0 ref_num:R
app_id:1936 pgm_num:0 branch:TBNY ord_num:F016 deal_number:0 ref_num:R
Q6I) deal_number:0 m:0 branch:NY ord_num:0440000010(
/sandbox]$>
6

This should remove all non-ASCII characters:

tr -d '\200-\377' < inputfile > outputfile

outputfile cannot be the same as inputfile.

7

I don't think nawk supports that character class.

Use /usr/xpg4/bin/awk instead:

/usr/xpg4/bin/awk '{gsub(/[[:cntrl:]]/,x)}1' bin4lines.txt
8

I'm not sure it matters, since these don't look like control characters to me.

9

found this with more googling

perl -pi2 -e 'BEGIN{$rep{chr($_)}=q() for 0..31,127;$rep{chr(10)}=chr(10)}s/([[:cntrl:]])/$rep{$1}/g'

this seems to work but not sure what this does though :slight_smile:
credit to http://compgroups.net/comp.lang.perl.misc/strip-control-characters-in-a-file/12755
---------- Post updated at 01:14 PM ---------- Previous update was at 12:49 PM ----------

@Corona688 tr did not work in this case

sandbox]$>tr -d '\200-\377'  < bin4lines.txt
app_id:1936 pgm_num:0 branch:TBNY ord_num:0500012(QMK) deal_num:0 ref_num:R
app_id:1396 pgm_num:0 branch:TBNY ord_num:0000016(�11S) deal_num:0 ref_num:R
app_id:1996 pgm_num:0 branch:TBNY ord_num:0260012(�PAK) deal_num:0 ref_num:R
app_id:1936 pgm_num:0 branch:TBNY ord_num:F016 deal_number:0 ref_num:R
added the line with issue here
Q6I) deal_number:0

@yoda
sandbox]$>awk '{gsub(/[[:cntrl:]]/,x)}1' < bin4lines.txt awk: syntax error near line 1
awk: illegal statement near line 1

Am on a solaris box

10

I said to run /usr/xpg4/bin/awk not awk

awk is broken on SunOS / Solaris.

11

Is there a way to change this so that it does not chnage the same file but writes into a new file instead of changing the same file

perl -pi2 -e 'BEGIN{$rep{chr($_)}=q() for 0..31,127;$rep{chr(10)}=chr(10)}s/([[:cntrl:]])/$rep{$1}/g'

---------- Post updated at 01:50 PM ---------- Previous update was at 01:46 PM ----------

@yoda this works when I use the awk you suggested

sandbox]$>/usr/xpg4/bin/awk '{gsub(/[[:cntrl:]]/,x)}1' < bin4lines.txt
app_id:1936 pgm_num:0 branch:TBNY ord_num:0500012(QMK) deal_num:0 ref_num:R
app_id:1396 pgm_num:0 branch:TBNY ord_num:0000016(�11S) deal_num:0 ref_num:R
app_id:1996 pgm_num:0 branch:TBNY ord_num:0260012(�PAK) deal_num:0 ref_num:R
app_id:1936 pgm_num:0 branch:TBNY ord_num:F016 deal_number:0 ref_num:R
added the line with issue here
app_id:1996 pgm_num:0 branch:TBNY ord_num:0440010(Q6I) deal_number:0

---------- Post updated at 01:54 PM ---------- Previous update was at 01:50 PM ----------

combining @rtrdx1 and @yoda I get this

sandbox]$>/usr/xpg4/bin/awk '{gsub(/[[:cntrl:]]/,x)}1' < bin4lines.txt | perl -pe 's/[^[:ascii:]]//g'
app_id:1936 pgm_num:0 branch:TBNY ord_num:0500012(QMK) deal_num:0 ref_num:R
app_id:1396 pgm_num:0 branch:TBNY ord_num:0000016(11S) deal_num:0 ref_num:R
app_id:1996 pgm_num:0 branch:TBNY ord_num:0260012(PAK) deal_num:0 ref_num:R
app_id:1936 pgm_num:0 branch:TBNY ord_num:F016 deal_number:0 ref_num:R
added the line with issue here
app_id:1996 pgm_num:0 branch:TBNY ord_num:0440010(Q6I) deal_number:0
12

So am I correct in assuming that you got the desired output?

If yes, you can redirect the output to another file to have a new file created rather than modifying the original file.

13

@yoda
yes thats what I ended up doing thanks