how to get rid of blank line in a flat text file

Shell Programming and Scripting
1

Hi, I have a flat text file which contains blank line between each text line. Is there any command to get rid of it?

Thanks for your help

2

You can use awk:

awk '$1 !~ /(^$)/' someFile > someNewFile
mv someNewFile someFile

Or you can use sed:

sed '/^$/ d' someFile > someNewFile
mv someNewFile someFile

And here's a script copied from the man page for sed:

In this example, sed removes all blank lines (including those with just
<Tab> and <Space> characters) from padded_file:

   sed '
   /^$/ d
   /^[<Tab><Space>]*$/ d
   ' padded_file
3

Thanks a lot oombera. I tested it out, it works fine. I have another question, in the text file, if the first character in each line starts with "E", then get rid of the whole line.

Thanks

Thanks, I got the result I want. But I didn't quite understand what ^$ means.

4

You can use either of the first two scripts I posted .. just replace ^$ with ^E

5

To reply to the edit you made in your post, when using commands like sed and awk that search for patterns, there are special characters, such as ^ (which represents the beginning of a line) and $ (which represents the end of a line).

Basically, whereas ^E searches for any line with an E immediately following the beginning of the line, ^$ searches for any line whose end immediately follows its beginning (a blank line).

6

Thanks oombera, I learned a lot from you today.

7

You can use grep command also for the same
grep -v "^$" someFile > someNewFile

Regards,
Yeheya

8

Here's a streamlined awk version:

awk '/^E/ {next}
NF > 0'

The default action in awk is print, so a pattern with no action just prints the line. The first pattern/action pair skips lines that start with "E". Since awk does an autosplit into $1, $2, ... on each input line after trimming leading and trailing whitespace (with the default delimiter /[ \t]+/), if there are only spaces, there are no fields, i.e., NF == 0 is true.

Note that after closing an action, the awk parser is in a "pattern state", so the above could be written on one line:

awk '/^E/ { next } NF > 0'

or

awk 'NF > 0 || !/^E/'

Notice in the latter that the easier condition is checked first, i.e., the regexp pattern match is only done if there are non-space characters on the line. With a huge file (or repeated execution) this means the latter should perform better (this claim is not tested).

9

Thank you all for your help. Does anyone can recommand some books for shell scripting?

Thanks

10

This forum is a great start. Use the search function of this site as well - you'll find this question has been asked many times..

11

Now I have two flat text files, I only need to choose some of the fields from both files and combine into one file, then load to a database table. Is there any way to do it?

Thanks for your help.

12

You'll need to be a bit more descriptive with what you're trying to do here.. i.e. what lines inparticular are you going to want to pull out? Read up on the grep command. If you're looking for lines with the word "hi" in them, you'd use something like grep "hi" file1 file2 > new_file.