Delete line with match and previous line quoting/escaping problem

Hi folks,

I've list of LDAP records in this format:

cat cmmac.export.tmp2
dn: deviceId=0a92746a54tbmd34b05758900131136a506,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:13:11:36:a5:06
dn: deviceId=0a92746a62pbms4662299650015961cfa23,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:15:96:1c:fa:23
dn: deviceId=0a92746a62sbms4662373470015961d6d7b,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:15:96:1d:6d:7b
dn: deviceId=0a92746a64cbms466298898001596010372,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:15:96:01:03:72
dn: deviceId=0a92746a64jbmr22a2083670015965883bc,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:15:96:58:83:bc

Some of these records have cmmac attribute duplicity, so I don't want to process them further. I have list of duplicate records, so I'm going through source file and deleting cmmac: MATCH and dn: above. I've tried to utilize solution I've found on these forums

grep -v "$(grep -B 1 "MATCH" cmmac.export.txt)" cmmac.export.txt > cmmac.export.filtered

Everything would be fine, but bash is developing the command into

grep -v '"$(grep -B 1 "MATCH" cmmac.export.txt)"' cmmac.export.txt > cmmac.export.filtered which is not producing desired result.

Thank you for any ideas how to solve this.
Tomas

#--------------------------------------------------------
#Filtering duplicities from device exports
#--------------------------------------------------------

cp cmmac.export.tmp cmmac.export.tmp2
for fil in `grep cmmac cmmac.export.tmp | sort | uniq -c | grep -v " 1 " | awk -F' ' '{print $3}'`
do
echo $fil
grep -v \""\$(grep -B 1 \" $fil\" cmmac.export.tmp2)"\" cmmac.export.tmp2 > cmmac.export.txt
cp cmmac.export.txt cmmac.export.tmp2
done

+ cp cmmac.export.tmp cmmac.export.tmp2
++ grep cmmac cmmac.export.tmp
++ sort
++ uniq -c
++ awk '-F ' '{print $3}'
++ grep -v ' 1 '
+ for fil in '`grep cmmac cmmac.export.tmp | sort | uniq -c | grep -v " 1 " | awk -F'\'' '\'' '\''{print $3}'\''`'
+ echo 00:15:d0:00:a5:e6
00:15:d0:00:a5:e6
+ grep -v '"$(grep -B 1 " 00:15:d0:00:a5:e6" cmmac.export.tmp2)"' cmmac.export.tmp2
++ bashtrap
++ echo 'CTRL+C Detected '

---------- Post updated 13-01-12 at 05:52 AM ---------- Previous update was 12-01-12 at 01:10 PM ----------

Somehow I've found what to do:
Grep doesn't like putting the variable from "for" cycle, so I've wrapped it into echo call in subshell. I've no idea why it works this way.

for fil in $( grep cmmac cmmac.export.tmp | sort | uniq -c | grep -v " 1 " | awk -F' ' '{print $3}' );
do      
#echo $fil
#Following line deletes line with pattern fil from file tmp2 and stores in txt
grep -v "$(grep -B 1 " `echo $fil`" cmmac.export.tmp2)" cmmac.export.tmp2 > cmmac.export.txt
cp cmmac.export.txt cmmac.export.tmp2
done

Please give an exemple of the wanted output

---------- Post updated at 12:22 PM ---------- Previous update was at 12:04 PM ----------

awk '{y=x;x=$0}$0~/cmmac/&&!A[$2]++{print "This record will be processed :" RS y RS $0}' yourinputfile
$ cat tst
dn: deviceId=0a92746a54tbmd34b05758900131136a506,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:13:11:36:a5:06
dn: deviceId=0a92746a54tbmd34b05758900131136a506,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:13:11:36:a5:06
dn: deviceId=0a92746a62pbms4662299650015961cfa23,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:15:96:1c:fa:23
dn: deviceId=0a92746a54tbmd34b05758900131136a506,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:13:11:36:a5:06
dn: deviceId=0a92746a62pbms4662299650015961cfa23,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:15:96:1c:fa:23
dn: deviceId=0a92746a64jbmr22a2083670015965883bc,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:15:96:58:83:bc
dn: deviceId=0a92746a62sbms4662373470015961d6d7b,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:15:96:1d:6d:7b
dn: deviceId=0a92746a64jbmr22a2083670015965883bc,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:15:96:58:83:bc
dn: deviceId=0a92746a64jbmr22a2083670015965883bc,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:15:96:58:83:bc
dn: deviceId=0a92746a64jbmr22a2083670015965883bc,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:15:96:58:83:bc
$ awk '{y=x;x=$0}$0~/cmmac/&&!A[$2]++{print "This record will be processed :" RS y RS $0}' tst
This record will be processed :
dn: deviceId=0a92746a54tbmd34b05758900131136a506,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:13:11:36:a5:06
This record will be processed :
dn: deviceId=0a92746a62pbms4662299650015961cfa23,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:15:96:1c:fa:23
This record will be processed :
dn: deviceId=0a92746a64jbmr22a2083670015965883bc,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:15:96:58:83:bc
This record will be processed :
dn: deviceId=0a92746a62sbms4662373470015961d6d7b,ou=devices,ou=customer,ou=nl,o=upc
cmmac: 00:15:96:1d:6d:7b

---------- Post updated at 12:33 PM ---------- Previous update was at 12:22 PM ----------

if you are just interested in extracting the cmmac with unicity, you can go with :

$ awk '/cmmac/&&!A[$2]++{print$2}' tst
00:13:11:36:a5:06
00:15:96:1c:fa:23
00:15:96:58:83:bc
00:15:96:1d:6d:7b

(just replace "tst" with the name of your input file)

1 Like

Hi ctsgnb,

what I've tried to achieve was:
I have a list of MAC's with duplicity in file in format:

00:15:96:1c:fa:23
00:15:96:1c:fa:24

I have a list of DN's and MAC's in format:

dn: deviceId=0a92746a54tbmd34b05758900131136a506,ou=devices,ou=customer,ou=nl,o=upc 
cmmac: 00:13:11:36:a5:06 
dn: deviceId=0a92746a62pbms4662299650015961cfa23,ou=devices,ou=customer,ou=nl,o=upc 
cmmac: 00:15:96:1c:fa:23 
dn: deviceId=0a92746a64jbmr22a2083670015965883bc,ou=devices,ou=customer,ou=nl,o=upc 
cmmac: 00:15:96:58:83:bc

So I need to search against MAC addresses in duplicity list, find it in list of DN and MAC and delete also the previous line with DN. My problem was construction of special grep command which has in "for .. in do .. done" cycle a fundamental flaw - it doesn't like just variable like %f, you need to utilize subshell - `echo %f ` to ensure that in expands into desired form. I've found a working solution

cp cmmac.export.tmp cmmac.export.tmp2 
for fil in $( grep cmmac cmmac.export.tmp | sort | uniq -c | grep -v " 1 " | awk -F' ' '{print $3}' ); do       
#echo $fil 
#Following line deletes line with pattern fil and preceding line from file tmp2 and stores in txt 
grep -v "$(grep -B 1 " `echo $fil`" cmmac.export.tmp2)" cmmac.export.tmp2 > cmmac.export.txt 
cp cmmac.export.txt cmmac.export.tmp2 
done

and it's output is:

dn: deviceId=0a92746a54tbmd34b05758900131136a506,ou=devices,ou=customer,ou=nl,o=upc 
cmmac: 00:13:11:36:a5:06 
dn: deviceId=0a92746a64jbmr22a2083670015965883bc,ou=devices,ou=customer,ou=nl,o=upc 
cmmac: 00:15:96:58:83:bc

Thanks for your interest :b:

You could try something like this:

awk 'NR==FNR{A[$1]=1;next}/cmmac/&&!A[$2]{print p RS $0}{p=$0}' file1 file2

or

awk 'NR==FNR{A[$1]=1;next}{p=$0;getline}!A[$2]{print p RS $0}' file1 file2
1 Like

Who has no idea what Mr. Scrutinizer wrote, see these articles:

unstableme.blogspot.com/search/label/awk%20FNR