Exclude variable from a file

Shell Programming and Scripting
1

Hello,

I have a list of patterns which I'd like to exclude from a file. I tried to do

while read line
do
grep -v $line file >> new.file
done < mylist

But obviously it won't work. It only works for just grep. I know I can do:

grep -Ev 'pattern1|pattern2|...' myfile

but my list is very long. Is there any other elegant way to do it? I don't need to use grep, anything else would be also fine.

Thank you!

2

Did you consider using a pattern file:

grep -vf mylist file

This may not be the most efficient method, though.

1 Like
3

Do you want to exclude the lines containing one of the patterns OR just the patterns in the lines?

4

I need to exclude the lines

5

Then RudiC's suggestion should do what you want...
If the patterns in mylist are fixed strings, then:

grep -Fvf mylist file

will run a little bit faster. And, if the patterns are extended regular expression instead of basic regular expressions or fixed strings, you'd need:

grep -Evf mylist file
1 Like
6

In this case go with RudiCs excellent suggestion: "-v" inverts the match, excluding the lines matching the pattern and "-f" reads the patterns from a file instead of arguments from the command line. His command will process all the patterns in one pass.

I hope this helps.

bakunin

PS: Darn, Don types fast. ;-))

1 Like
7

Thank you very much! It's working :slight_smile:

8

Can I add a word of caution that the lines in your exclude file will match anywhere on the line in your input file. If you need to ensure that it is (for example) only lines starting with abc then your exclusion file should be written as ^abc

So, for the input file:-

abc123
def456
123abc
456def
The end

... and the exclusion file:-

abc

you would get just:-

def456
456def
The end

If you exclusion file is

^abc

... then the output would be:-

123abc
def456
456def
The end

If you want to exclude lines with the listed patterns in other specific places then there are usually ways to do that too.

I hope that this helps avoid a problem (if it even exists)
Robin

1 Like