issue with ^B characters in the file

Shell Programming and Scripting
1

Hello-

I have a csv file with japan email ids as the last column(pipe delimited file).
One of the email ids has some special characters which are not displayed on my UNIX box, however when ftped to local(Windows machine) I can see the email id terminated with a square box instead of pipe.

How can I remove these records from the csv file on the unix box?

Sample Data:
123456|23456|432|21|12|ABC-BCE| |N|email@email.ne

Thanks
Raghs....

2

You can try something like:

sed 's/..$//' file > newfile

Regards

3

We are building a collection of unix and MSDOS methods for identifying non-display characters on another post. Once you know what is in that byte it is easier to make a match with sed, grep or whatever.

http://www.unix.com/shell-programming-scripting/116557-ascii-code.html

Not clear whether you wanted to lose the whole line containing the unusual character, or just the unusual character.

Beware that text mode FTP will strip the high order bit such that the character can change between unix and MSDOS.