Find matched patterns in multiple files

Shell Programming and Scripting
1

Hi,

I need help to find matched patterns in 30 files residing in a folder simultaneously. All these files only contain 1 column. For example,

File1

Gr_1
st-e34ss-11dd
bt-wwd-fewq
pt-wq02-ddpk
pw-xsw17-aqpp
Gr_2
srq-wy09-yyd9
sqq-fdfs-ffs9
Gr_3
etas-qqa-dfw
ddw-ppls-qqw
pft-78bn-htw0

File2

Gr_1
swq-ws09-fkd9
lpt-ddhg7-jjdk
klp-hdh76-ddl
Gr_2
etas-qqa-dfw
hfjs-90kj-hjs6
lks-gg77-jklo
Gr_3
pld-sgh09-qqw
dgf-23erf-we4
yur-aa23-fdfw
qoe-ewd8-kd9
Gr_4
st-e34ss-11dd
pw-xsw17-aqpp
pji-98hj-mkl

File3

Gr_1
pw-xsw17-aqpp
sad-sw09-dff
tdh-2kd-saww
add-aww2-qw9
Gr_5
swq-ws09-fkd9
hdjs-ss9-eew
lp0-89ks-2ww
Gr_10
ddf-da9-wwq
ffdw-2q1s-df
st-e34ss-11dd

The output file should be like below. It should print only the matched patterns (in blue) from those files:

outfile.txt

st-e34ss-11dd	Gr_1:File1	 Gr_4:File2	Gr_10:File3
pw-xsw17-aqpp	Gr_1:File1	 Gr_4:File2	Gr_1:File3
swq-ws09-fkd9	Gr_1:File2	 Gr_5:File3
etas-qqa-dfw	Gr_3:File1	 Gr_2:File2

I did the following codes. but this codes does not give me the results that i wanted as above. It did not give me the "Gr_x".

grep -w -f * >outfile.txt

Appreciate if someone could help me. Thanks.

2 
awk 'NR==FNR {a[$0]=$0;next}
	 /^Gr_[0-9]+$/ {key=$0} 
	 ($0 in a) { B[a[$0]]=B[a[$0]] "\t" key":"FILENAME} 
	 END { for (i in B){print i FS B}
		}' ptr1 file[123]

This produces

pw-xsw17-aqpp   Gr_1:file1        Gr_4:file2        Gr_1:file3
st-e34ss-11dd   Gr_1:file1        Gr_4:file2        Gr_10:file3
etas-qqa-dfw    Gr_3:file1        Gr_2:file2
swq-ws09-fkd9   Gr_1:file2        Gr_5:file3

The ptr1 file contains the patterns you are looking for (the blue's) like this

$ cat ptr1
etas-qqa-dfw
pw-xsw17-aqpp
st-e34ss-11dd
swq-ws09-fkd9

Note: Look for the ^Gr_[0-9]+$ carefully and modify it accordingly if it doesn't match with your real data.

3

Hi clx,

i tried your code but it gives me syntax error at line 2. :

awk: line 2: syntax error at or near ^
awk: line 2: runaway regular expression /{key=$0}

I understand your not there. as for the above syntax error, i got it when i ran the above sample input files.

Also, for my real input files, it does not mean that all the file name start with "File" and then followed by numbers. It could be any random names. thanks

4

Which OS are you working on. If its SunOS, use nawk or /usr/xpg4/bin/awk

Naming of files is not an issue. You can provide wildcard also. Anything which you can list

*.txt 
*.csv
file_*.txt 
file1 tx.txt
5

i am using linux (ubuntu). as for the file names, ok, i will use the wildcard. thanks

6

Can you show the exact command you are trying?

7

i am using the one that u gave me as i am testing on the sample input files that i gave.

awk 'NR==FNR {a[$0]=$0;next}      
/^Gr_[0-9]+$/ {key=$0}       
($0 in a) { B[a[$0]]=B[a[$0]] "\t" key":"FILENAME}       
END { for (i in B){print i FS B}         
}' ptr1 file[123]
8

I am not sure, what went wrong. Its working for me. Can you try putting all in single line?

awk 'NR==FNR {a[$0]=$0;next} /^Gr_[0-9]+$/ {key=$0} ($0 in a) { B[a[$0]]=B[a[$0]] "\t" key":"FILENAME} END { for (i in B){print i FS B} }' ptr1 file[123]

Again, Make sure you have ptr1 file and file1,file2,file3 (lower-case)

This gives me :

pw-xsw17-aqpp   Gr_1:ff1        Gr_4:ff2        Gr_1:ff3
st-e34ss-11dd   Gr_1:ff1        Gr_4:ff2        Gr_10:ff3
etas-qqa-dfw    Gr_3:ff1        Gr_2:ff2
swq-ws09-fkd9   Gr_1:ff2        Gr_5:ff3
1 Like
9

My bad, i did not create ptr1. it works now but the results is not accurate. I got this as my output:

st-e34ss-11dd 	Gr_4:file2	Gr_10:file3
Gr_1 	Gr_1:file2	Gr_1:file3
Gr_2 	Gr_2:file2
Gr_3 	Gr_3:file2
etas-qqa-dfw 	Gr_2:file2
pw-xsw17-aqpp 	Gr_4:file2	Gr_1:file3

some are missing, like

swq-ws09-fkd9   Gr_1:file2       Gr_5:file3

and

Gr_1:file1

for pw-xsw17-aqpp. Also, as u can see above, line 2-4 is weird. thanks

10

If what you're trying to do is just to print strings (other than Gr_digits ) that appear in more than once in your input files, try:

awk '
/^Gr_/ {group = $1
	next
}
{	c[$1]++
	d[$1] = d[$1] "\t" group ":" FILENAME
}
END {	for(i in c)
		if(c > 1)
			printf("%s%s\n", i, d)
}' File*

which, with File1, File2, and File3 as shown in the 1st message in this thread, produces:

st-e34ss-11dd	Gr_1:File1	Gr_4:File2	Gr_10:File3
swq-ws09-fkd9	Gr_1:File2	Gr_5:File3
etas-qqa-dfw	Gr_3:File1	Gr_2:File2
pw-xsw17-aqpp	Gr_1:File1	Gr_4:File2	Gr_1:File3
1 Like
11

Hi Don Cragun,

Awesome!!! I run the codes on my real data and it works perfectly. Thanks a million. :slight_smile: