Remove lines if the first character is "|"

UNIX for Advanced & Expert Users
1

Hi.

I have a huge file (350 million lines).
I need to delete all lines in it that:

  1. Begin with a pipe character -- '|'
  2. Or have less than 5 pipe characters in the line

Have been searching for some SED/AWK help for this (which is faster btw on such a ginormous file?) but only found this, but this works with a number not a pipe character.

On this website: 
 /unix-advanced-expert-users/160025-deleting-lines-if-ith-character-x.html

Much appreciate any pointers.

2

Does not matter - the time is due to I/O. sed reads the file once as does awk.

awk -F '|'  ' /^|/ || NF<5 {next} {print } ' bigfile > newbigfile
3

Thanks Jim. Doesn't work. The "newbigfile" is totally blank.

---------- Post updated at 07:58 AM ---------- Previous update was at 07:28 AM ----------

Found it. The first | has to be escaped by a back slash. So the command should be:

awk -F '|'  ' /^\|/ || NF<5 {next} {print } ' bigfile > newbigfile
4

Try this

awk -F "|" '{if( NF>5 && $1 != "") print}' inputfile
5

Hi Sorry to intrupt in this thread.
i am new to this form..
can any one please let me know how to create new thread?

6

Go to the relevant forum and click on New Thread on top left of the forum.

1 Like
7

Thanks tene

8

An infile editor:

perl -F\| -lani -e 'print unless ( scalar(@F)<6 ||/^\|/)' bigfile