Trying to remove '^M' characters from a file.

Hi guys,

Hope you are all well.

This is a line of data from a csv file. I have used vi and set the 'set list' option to display the trailing $ character.

"01","Grocery","01006","eat Fish & Spreads"$

I have tried the following commands, but neither of them appear to be working?

1) tr -d "^M" < originalfile.csv > newfile.csv
2) tr -d "\015" < originalfile.csv > newfile.csv

....but when I vi & set list the new file, I am still seeing the trailing $ character in the file.

Can anyone please tell me where I am going wrong?

Thanks in advance,
Kev

Hope this helps.

I had a big issue with this while trying to get a flat file into Oracle. I tried using vi and sed, and ultimately came upon a solution using strings and tr.

Here is my shell script:

#make a backup
cp $1 $1.orig

#Strip out nulls.
tr -d �\0 < $1.orig > $1.temp

#Strip out other unprintable characters
strings $1.temp > $1

#Remove temp file.
rm $1.temp

#Fix permissions. (You may not need this for your application)
chmod 744 $1

Have you misunderstood vi's "set list". What it does is print a "$" sign at the end of each line (and show tabs as ^I). The $ sign isn't in the file itself it is just displayed to make it easier for you to spot things like trailing spaces at the end of lines.

cheers

The "^M" things are from dos/windows files. dos carriage returns are "\n\r", unix "\n"

The extra character shows up as "^M".

Use dos2ux (or dos2unix depending on your flavor of unix) to convert windows text files.

dos2ux oldfile > newfile

in vi you can replace all ^M characters with command
:%s/^M//g

(Note that to type ^M in vi type Ctrl-V Enter)

I realise that this note may be offensive for vi users

Thanks for the swift response guys - much appreciated.

Thestevew was right - I was misunderstanding the vi 'set list' option, so thanks for putting me straight.

I eventually came up with a couple of possibilites - I could use the unix2dos command within a batch file to properly format my csv file.

I also found another thread on this forum with the following perl command...

perl -i -pe 's/$/\r/' myfile.csv

I'll probably use the perl command to avoid implementing a batch file. Plus, I don't have to output the results to a new file.

Out of interest, does anyone know of a sed command that would do the same as the above perl command?

Thanks again,
Kev

sed -i 's/$/\r/' myfile.csv

I don't think that all implementations of sed has this feature. I've used GNU sed 4.1.4

This is a very small issue but create big problems in unix.I have faced this issue several times and found dos2unix command most easiest way.Try this command

dos2unix filename filename

keeping same filename will remove the ^M character from file.

Hi,

You can use below sed command to remove the ^M characters.

sed 's/.$//' filename > temp
mv temp filename

Helo.

sed 's/^M//g' input_file > output_file

will do the job, but to get the "correct" "^M" use:
"control+v" followed by "control+m" key combinations.

Regards.

^M must be removed from the end of line, so

$ sed 's/^M$//' input_file > output_file

Yes, Hitori, you are right. I forgot to add de "$"
My sed command would remove ^M wherever it is. The "g" is not necessary because it means "in the whole line".
I'm sorry.

Thanks so much for mentioning the <Ctrl v + Enter>. Many people leave out the <Enter> and it doesn't work for dummies. Just simple and clear

I was in big shit but am good now....

Cheers!!!!!!!!!!!!!!!!!!!!!

i have the same problem. tried to do it with sed, but got no luck.
so i found my way to remove those ^M

cat -A input_file | tr -d ^M$ > output_file

works nice on my Debian.