Tried many options but unable to delete blank lines from text file

Hi,

I tried the following options but was unable to delete blank lines from file

Input file = temp.hash.txt
temp.hash.txt content

90

0

89.56
0
0


57575.4544
56.89




0


0
0
0

----------------------------
Options tried

  1. sed '/^$/d' temp.hash.txt > temp.hash.out
  2. grep '[^[:blank:]]' < temp.hash.txt > temp.hash.out
  3. awk 'NF' temp.hash.txt
  4. sed -i '/^$/d' temp.hash.txt
  5. grep -v '^$' temp.hash.txt > output.txt

Nothing worked.

Any help is appreciated!

Thanks...

Any of your options tried does what you want them to do, so I guess your file has some "hidden" features like e.g. non-printing control characters. Try

od -tx1c temp.hash.txt

and post the result.

This is the result

[oratest@uswclora15 ~]$ od -tx1c temp.hash.txt
0000000 39 30 0d 0a 0d 0a 30 0d 0a 0d 0a 38 39 2e 35 36
          9   0  \r  \n  \r  \n   0  \r  \n  \r  \n   8   9   .   5   6
0000020 0d 0a 30 0d 0a 30 0d 0a 0d 0a 0d 0a 35 37 35 37
         \r  \n   0  \r  \n   0  \r  \n  \r  \n  \r  \n   5   7   5   7
0000040 35 2e 34 35 34 34 0d 0a 35 36 2e 38 39 0d 0a 0d
          5   .   4   5   4   4  \r  \n   5   6   .   8   9  \r  \n  \r
0000060 0a 0d 0a 0d 0a 0d 0a 30 0d 0a 0d 0a 0d 0a 30 0d
         \n  \r  \n  \r  \n  \r  \n   0  \r  \n  \r  \n  \r  \n   0  \r
0000100 0a 30 0d 0a 30 0d 0a 0d 0a 0d 0a 35 37 35 36 35
         \n   0  \r  \n   0  \r  \n  \r  \n  \r  \n   5   7   5   6   5
0000120

Hello uuuunnnn,

So it is confirmed now that \r carriage characters are present in your Input_file, so could you please first remove them and then try your any of the code mentioned in POST#1 and let us know how it goes then.

tr -d '\r' < Input_file > temp_Input_file  &&  mv temp_Input_file  Input_file

Thanks,
R. Singh

You can also use grep but with [[:space:]] instead of [[:blank:]] :

grep -v '^[[:space:]]*$'  file

But this will leave the Windows carriage return character on the other lines.

--
You could do it all at once like so:

awk '{sub(/\r/,x)}NF' file

I did that, but it completely deleted all lines

[oratest@uswclora15 ~]$ tr -d '\r' < temp.hash.txt > temp_Input_file  &&  mv temp_Input_file  temp.hash.txt
[oratest@uswclora15 ~]$ vi temp.hash.txt
[oratest@uswclora15 ~]$ sed '/^$/d' temp.hash.txt > temp.hash.txt
[oratest@uswclora15 ~]$ vi temp.hash.txt
[oratest@uswclora15 ~]$

Thanks,
Niladri

Whoa!! I hope you have a backup of your file. :slight_smile:

You are essentially trying to read from a file, make some changes and write back to it. It doesn't work the way it looks. (*)

A few options are:
1) If your sed version allows it, try the "in-place edit" modifier; something like "sed -i". Check your sed manual "man sed".

2) Use a temporary file as you did for the "tr" command.

======
(*)
As for why it doesn't work the way it looks - you have two different processes (a) a read process and (b) a write process, both sharing the same resource - your file. When they are executed by the OS, a race condition occurs between them which renders the final output unpredictable.
If the writing is done before the reading, you get a zero-byte file. If the writing is done mid-way during the reading process, you may get a "half-baked" output file. The safest way, therefore, is to either use a temporary file explicitly or let sed do it via the "-i" modifier - which is not available in all sed versions.
You could also use other tools - Bash, awk, Perl etc. but the temporary file will be involved, either explicitly or implicitly.

Hello durden_tyler/uuuunnnn,

That tr command worked for me. Also && condition will make sure like first reading of Input_file and it's output's redirection is done then only it will go to next command which is mv temp_Input_file Input_file , ideally it shouldn't fail. Kindly do let me know your views on same.

Thanks,
R. Singh

The explanation was for the part in red in my post; I thought the post was clear about the erroneous part.

1 Like

Thanks everyone.

Yes the issue was that I was copying the values from windows to unix and carriage return was entered. As suggested I entered the values in unix and used

sed '/^$/d' temp.hash.txt > temp.hash.out

It worked fine.

Thanks