Grep Problem

UNIX for Beginners Q & A UNIX for Dummies Questions & Answers
1

Dear all,
my question is about grep.

i have two files.

the first one is a list with ids (line number 188) like this:

comp10045_c0_seq1
comp10118_c0_seq1
comp10221_c0_seq2
comp10503_c0_seq1
comp10504_c0_seq1
comp10504_c0_seq2
comp10631_c0_seq1
comp10631_c0_seq2
comp10646_c0_seq1
comp10804_c0_seq1

the second file is a tab seperated table (line number 329) , like this:

comp10045_c0_seq1    tr|Q6PBS3|Q6PBS3_DANRE    79.31    464    93    1    2024    633    2    462    0.0     776
comp10118_c0_seq1    tr|Q6P0C4|Q6P0C4_DANRE    44.19    43    23    1    1066    1194    177    218    0.29    35.4
comp10221_c0_seq2    tr|Q7T340|Q7T340_DANRE    95.05    303    15    0    175    1083    1    303    0.0     543
comp10503_c0_seq1    tr|F1Q9U6|F1Q9U6_DANRE    69.07    472    142    2    2330    915    1    468    0.0     687
comp10504_c0_seq1    tr|F8W2T4|F8W2T4_DANRE    98.01    151    3    0    787    1239    6    156    2e-99     307
comp10504_c0_seq2    tr|F8W2T4|F8W2T4_DANRE    98.01    151    3    0    1095    1547    6    156    2e-98     307
comp10631_c0_seq1    tr|B0S5C5|B0S5C5_DANRE    90.61    394    37    0    145    1326    1    394    0.0     687
comp10631_c0_seq2    tr|B0S5C5|B0S5C5_DANRE    90.61    394    37    0    145    1326    1    394    0.0     687
comp10646_c0_seq1    tr|F1QDS2|F1QDS2_DANRE    84.82    850    110    7    173    2695    1    840    0.0    1333
comp10804_c0_seq1    tr|Q7ZUP6|Q7ZUP6_DANRE    93.58    452    26    2    156    1511    1    449    0.0     851
comp10991_c1_seq1    tr|E7F8T6|E7F8T6_DANRE    94.21    484    28    0    1974    523    9    492    0.0     943

i want to grep the lines of the second file which have the same ids of the first file using the following code:

 grep -f file1 file2 > output

but it does not work cause i have 329 lines instead of 188.

Can anybody explain what i am doing wrong?

Regards

2

Hello kleinstadt,

Welcome to forums, thank you for using code tags in your posts for commands.
Following may help you in same.

awk 'FNR==NR{X[$1]=$1;next} ($1 in X){print $0}' file1 file2

Output will be as follows.

comp10045_c0_seq1    tr|Q6PBS3|Q6PBS3_DANRE    79.31    464    93    1    2024    633    2    462    0.0     776
comp10118_c0_seq1    tr|Q6P0C4|Q6P0C4_DANRE    44.19    43    23    1    1066    1194    177    218    0.29    35.4
comp10221_c0_seq2    tr|Q7T340|Q7T340_DANRE    95.05    303    15    0    175    1083    1    303    0.0     543
comp10503_c0_seq1    tr|F1Q9U6|F1Q9U6_DANRE    69.07    472    142    2    2330    915    1    468    0.0     687
comp10504_c0_seq1    tr|F8W2T4|F8W2T4_DANRE    98.01    151    3    0    787    1239    6    156    2e-99     307
comp10504_c0_seq2    tr|F8W2T4|F8W2T4_DANRE    98.01    151    3    0    1095    1547    6    156    2e-98     307
comp10631_c0_seq1    tr|B0S5C5|B0S5C5_DANRE    90.61    394    37    0    145    1326    1    394    0.0     687
comp10631_c0_seq2    tr|B0S5C5|B0S5C5_DANRE    90.61    394    37    0    145    1326    1    394    0.0     687
comp10646_c0_seq1    tr|F1QDS2|F1QDS2_DANRE    84.82    850    110    7    173    2695    1    840    0.0    1333
comp10804_c0_seq1    tr|Q7ZUP6|Q7ZUP6_DANRE    93.58    452    26    2    156    1511    1    449    0.0     851

Thanks,
R. Singh

3

Hi Singh,
thanks for your solution and fast reply. the code you postet works.
But why does grep not work with my files? i know it should work...

4

Hello kleinstadt,

You can check here about carriage charaters if they are there, by cat -v Input_file if they are present then you need to remove them. I tested the same command in my GNU grep and it worked fine for me, you can also check your grep version by grep --version too.

Thanks,
R. Singh

5

My first guess would be DOS <CR> line terminators. How were those files produced? If I add <CR> chars, the grep -f fails, without it works. RavinderSingh13's proposal will also fail with <CR>s present ( sub (/\r$/,"") could help in some awk s).
Remove them with e.g. tr -d '\r' <file1 or dos2unix file1 .

6

As suggested, your BLAST table and/or your sequence list may contain carriage returns, which can be removed by any of the above methods or

sed 's/\r//g' file

Also your grep would work, I suggest using Ravinders script or

grep -w

to avoid erroneous calls.

8

Could it be that grep has a problem if there are more lines with the same id in file 2?
Also i could not find any carriage characters...

9

Pls attach the original files so we can look into them.

10

The grep utility does not care if there are duplicates.

Show us the output from the command:

head file1 | od -bc

where file1 is the file containing the list of ids you mentioned in the 1st post in this thread.

11

Dear all,
thanks again for your kindly help!
Attached you can find the files which i am using at the moment.
The lociid file (=file1) was generated with the following steps:

cut -f1 table > loci
sort loci > loci_sorted
uniq loci_sorted > lociids

For uploading the files i had to "rename" them:

mv lociids lociids.txt
mv table table.txt

Here is the output of the

 head lociids.txt | od -bc

requested by Don Cragun

0000000 143 157 155 160 061 060 060 064 065 137 143 060 137 163 145 161
          c   o   m   p   1   0   0   4   5   _   c   0   _   s   e   q
0000020 061 012 143 157 155 160 061 060 060 071 061 137 143 060 137 163
          1  \n   c   o   m   p   1   0   0   9   1   _   c   0   _   s
0000040 145 161 061 012 143 157 155 160 061 060 061 061 070 137 143 060
          e   q   1  \n   c   o   m   p   1   0   1   1   8   _   c   0
0000060 137 163 145 161 061 012 143 157 155 160 061 060 061 065 061 137
          _   s   e   q   1  \n   c   o   m   p   1   0   1   5   1   _
0000100 143 060 137 163 145 161 061 012 143 157 155 160 061 060 061 065
          c   0   _   s   e   q   1  \n   c   o   m   p   1   0   1   5
0000120 061 137 143 060 137 163 145 161 062 012 143 157 155 160 061 060
          1   _   c   0   _   s   e   q   2  \n   c   o   m   p   1   0
0000140 061 065 071 137 143 060 137 163 145 161 061 012 143 157 155 160
          1   5   9   _   c   0   _   s   e   q   1  \n   c   o   m   p
0000160 061 060 061 071 070 137 143 060 137 163 145 161 061 012 143 157
          1   0   1   9   8   _   c   0   _   s   e   q   1  \n   c   o
0000200 155 160 061 060 062 060 064 137 143 060 137 163 145 161 061 012
          m   p   1   0   2   0   4   _   c   0   _   s   e   q   1  \n
0000220 143 157 155 160 061 060 062 060 064 137 143 060 137 163 145 161
          c   o   m   p   1   0   2   0   4   _   c   0   _   s   e   q
0000240 062 012 143 157 155 160 061 060 062 062 061 137 143 060 137 163
          2  \n   c   o   m   p   1   0   2   2   1   _   c   0   _   s
0000260 145 161 062 012
          e   q   2  \n
0000264
12 
grep -f lociids.txt table.txt

works as intended on two different systems, although extremely slowly on one of them. To speed it up, try

grep -Ff lociids.txt table.txt
13

When i try it there are again as many lines as in the table file...

14

You have 1955 duplicates in table.txt. Well possible that all are found in lociids.txt. Convince yourself - reduce lociids.tst to, say, 10 lines and try again...

15

i tried it with 10 lines but got 11 in my outputfile

16

Duplicates in Col 1?

17

I ran your files like the following

comm -13  <(grep -Ff lociids.txt table.txt | awk '{print $1}' | sort -u) <(sort lociids.txt)

If there was more output, it would have shown up, grep is doing exactly as intended

18

So instead of running:

grep -Ff lociids.txt table.txt

use:

grep -Fnf lociids.txt table.txt

which will give you the line numbers in table.txt along with the lines matched.

We all expect that two or more lines in the output will have been matched by the same id.

Show us the ten line lociids.txt you used and the eleven lines of output from get from:

grep -Fnf lociids.txt table.txt

(using CODE tags) and tell us which one of those eleven lines should not have been found by grep .

19

Hi,

Your problem with grep is a school case, by example, when you search:
comp30188_c3_seq1
you find also:
comp30188_c3_seq13
comp30188_c3_seq18

Regards.