Issue with Removing Carriage Return (^M) in delimited file

UNIX for Advanced & Expert Users
1

Hi - I tried to remove ^M in a delimited file using "tr -d "\r" and "sed 's/^M//g'", but it does not work quite well. While the ^M is removed, the format of the record is still cut in half, like
a,b, c
c,d,e

The delimited file is generated using sh script by outputing a SQL query result to tab delimited file. The ^M embedded in the record is captured from user input, I believe.

Any insight is appreciated.

2

Or at least the first few lines of data?

try something like the following which will look at the first five lines:
(use your file in place of infile)

>head -5 infile | od -An -t oC -w10

Note, in octal
<CR> carriage return = 015
<LF> line feed = 012

3

Hi joeyg,

Thanks for your reply. Unfortunately, this is sensitive data so I can't send out a dump. I did use the od command to pull the data and checked for \r (od -c). I see that there is carriage return (\r)+ new line control (\n) embedded in the record.

When I used tr -d "\r\n" , however it remove all line breaks with just "\n". Do you know how I can remove only the string "\r\n" without touching those with only "\n"?

Thanks,
sirahc

4

How about something like:

infile="original.dat"
outfile="revised.dat"
while read zf
  do
    printf "$zf\n" >>$outfile
done < $infile
5

how abt dos2ux / dtox command ?

6

Thanks for all your suggestions. :slight_smile: The printf/dos2unix commands did work to some extend but cause format problem to other columns. I eventually used SQL to remove ^M before outputting it to the file.

7

in vi:

# vi datafile
:s/^V^M//g
:wq
#

the ^V^M sequence means control-V control-M

8

Or in vim

:set fileformat=unix
:w