How to replace ^J in unix file

Shell Programming and Scripting
1

Hi,

i am facing issue while reading files in unix, which has ^J at end of every line. I want to know how to replace it in the file.

Example: infile.txt has below data
-----------------
20101030^J
110

Now in the above file i want unix command to replace the ^J character . so that i will use in my script for source files to replace the '^J' character.

I am New bee to Control Characters. Dont know how to deal with them. Please help me in fixing this.

Thanks very much in Advance.
KK:)

2

You can try

sed 's/\^J//g' inputfile
3

But the sed command not modifying the source file.. it is displaying the data without ^J character . but not modifying the source file.

4

This is a strange request because ctrl/j is line-feed is the standard unix line text file terminator.

What software are you using to see this character(s)?
Can you display the file in Hex or Octal so we can see the character(s)?

What do you get for this "sed" enquiry which is designed to make control characters visible:

sed -n l infile.txt

Sed will not work properly unless we have a correctly formatted text file.

5

Output it to some other file

sed 's/\^J//g' inputfile > outputfile
6

When dealing with characters, that are not easy to type in an editor, I usually find the "offending" character with od. od can be found on most *nixes, and supports multiple ways to display it's input.

Here are a few examples. First I created some arbitary data:

perl -e 'print chr($_) for (1..10);' > some_data

And the outputs of od for that file:

Hexadecimal 2-byte output:

$ od -h < some_data
0000000 0201 0403 0605 0807 0a09
0000012

Hexadecimal 2-byte output with the characters next to it:

$ od -hc < some_data
0000000 0201 0403 0605 0807 0a09
        001 002 003 004 005 006  \a  \b  \t  \n
0000012

Octal 2-byte output with the characters:

$ od -oc < some_data
0000000 001001 002003 003005 004007 005011
        001 002 003 004 005 006  \a  \b  \t  \n
0000012

Octal output with the characters:

$ od -bc < some_data
0000000 001 002 003 004 005 006 007 010 011 012
        001 002 003 004 005 006  \a  \b  \t  \n
0000012

Then just use sed or something similar to remove/translate the characters.

EDIT: Added last example.

7

I have done the file ftp and place it in AIX box. Now i am surprised with the source file... it is not showing any special character in the file when i opened through vi. but when i have given the command tail -1 input.txt.. it is not giving the last line , it is giving all the two lines from the source file.

file inut.txt:
-----------
20101021
1023

Below is the output after giving the command tail -1 input.txt:
20101021
1023

it is displaying all the lines from the source file :confused:. I am shocked tat why it is happening even though there is no special character visible in the file.
How to fix this. and identify the special character.
Please suggest how to fix it

Thanks In Advance
kk

8

Please post the answer to my questions from post #4 and the "od" output from post #6.
We won't get anywhere until we know what characters are in the file.

9

Thanks for reply... below are the results that i captured in AIX box for the file inputfile.txt

Post 4 output:
-------------
sed -n l inputfile.txt.txt
1231231231$
asdafsaf$

post 6 output:
------------------
$ od -h <inputfile.txt.txt
0000000 3132 3331 3233 3132 3331 0a61 7364 6166
0000020 7361 6600
0000023
$ od -hc <inputfile.txt.txt
0000000 3132 3331 3233 3132 3331 0a61 7364 6166
1 2 3 1 2 3 1 2 3 1 \n a s d a f
0000020 7361 6600
s a f
0000023
$ od -oc <inputfile.txt.txt
0000000 030462 031461 031063 030462 031461 005141 071544 060546
1 2 3 1 2 3 1 2 3 1 \n a s d a f
0000020 071541 063000
s a f
0000023
$ od -bc <inputfile.txt.txt
0000000 061 062 063 061 062 063 061 062 063 061 012 141 163 144 141 146
1 2 3 1 2 3 1 2 3 1 \n a s d a f
0000020 163 141 146
s a f
0000023

10

What you have there is not a text format file. There is no trailing line-feed character on the last record.

If (referring to post #1) you want to remove the rest of the carriage return characters to produce a file which might be suitable for your program but is unsuitable for processing in shell or common utilities such as "sed" "tail" etc.:

cat oldfile | tr -d '\n' > newfile
11

EDIT: already answered