I have a database of possible primary signal strings
pp22
pt22dx
pp22dx
jty2234
Also I have a list of scrambled signals which has a shorter string and a longer string separated by // (double slash ). Always the shorter string of a scrambled signal will have the primary signal as a substring and I need to parse out the longest possible primary signal from it.
is pp22dx instead of pp22 because pp22dx is longer. But why isn't pt22dx just as valid as pp22dx ? (Both appear in the input file named primary , both are the same length, and pt22dx appears on that line of input before pp22dx .)
Assuming either input is allowed, you could try something like:
awk -F// -v OFS="\t\t" '
FNR == NR {
if((l = length($0)) in pat) {
pat[l] = pat[l] "|" $0
next
}
pat[l] = $0
for(i = 1; i <= nl && lengths > l; i++)
new = lengths
new = l
for(; i <= nl; i++)
new[i + 1] = lengths
nl++
for(i = 1; i <= nl; i++)
lengths = new
next
}
{ for(i = 1; i <= nl; i++)
if(match($1, pat[lengths])) {
print $0, substr($1, RSTART, RLENGTH)
next
}
print $0, "No match"
}' primary scrambled
which with your sample inputs produces the output:
The above code just uses awk features defined by the standards. If someone wants to try this on a Solaris/SunOS system, change awk to /usr/pg4/bin/awk or nawk . Sorting the lengths array might be done more efficiently with built-in functions in some versions of awk .
only the shorter of the two columns (compared by length separated by //) will have the primary signal, in this case [(\pp22dx)-@@----B-@-- , so the answer is pp22dx .
Are you saying that what you really want is the longest string that matches an entire line in the file named primary that appears on both sides of the // on each line in the file named scrambled ?
To determine whether the time my script spends keeping the lengths[] array in order is worth the effort, how many lines are typically in each of your two real input files?
Not the entire the line, only the shorter split (separated by //) will have the primary signal.
In my original code I use
{ short=length($1)>length($2)? $2:$1 } to determine which of the two fields is shorter in length and just search that for the primary signal if(short~ps) .
There are 10k primary signals and 400k scrambled signals at this point. The scrambled signals will be more (~100k per year) as we collect more data.
With your sample data, the following seems to do what you want fairly efficiently:
awk -F// -v OFS="\t\t" '
BEGIN { M = 0
m = 32000
}
FNR == NR {
if((l = length($0)) in pat) {
pat[l] = pat[l] "|" $0
#printf("appending pattern: pat[%d]=\"%s\"\n", l, pat[l])
} else {pat[l] = $0
#printf("adding pattern: pat[%d]=\"%s\"\n", l, pat[l])
if(l > M)
M = l
if(l < m)
m = l
}
next
}
#FNR == 1 {
# for(i = M; i >= m; i--)
# if(i in pat)
# printf("pat[%d]=\"%s\"\n", i, pat)
#}
{ short = length($1) < length($2) ? 1 : 2
for(i = M; i >= m; i--)
if(i in pat && match($short, pat)) {
print $0, substr($short, RSTART, RLENGTH)
next
}
print $0, "No match"
}' primary scrambled
but I don't have any good way to test how this might work if you have EREs with thousands of patterns to be matched as alternatives in a single ERE. With your sample inputs, it produces:
as its output (which I think meets your requirements).
PS: Depending on the distribution of the lengths of your patterns, you might want to try the code I used in post #2 in this thread to sort the lengths and avoid processing lengths that don't exist in your data. I assume you won't have any problem merging the code from these two suggestions to get what you need.