Deleting lines from a stream after matching a pattern

Shell Programming and Scripting
1

Hi,

I have a requirement to to an ldapsearch and remove the shadow attributes in the output file.
What I do is ldapsearch() | operation to remove shadow > FILE
The ldapsearch gives output like this(with same line formation):

objectClass: FSConfig
objectClass: extensibleObject
fsCAIP: 1.1.1.1
fsCAPort-shadow: '<fsCAPort-shadow status="previous" currentValu
 e="ODA4MA==">ODA4MA==</fsCAPort-shadow>'
fsCAIP-shadow: '<fsCAIP-shadow status="previous" currentValue="M
 TAuOC4yMDcuNTQ=">MTAuOC4yMzIuMTcx</fsCAIP-shadow>'
fsCAPort: 8080
fsName: C=IN
fsName-shadow: '<fsName-shadow s
 tatus="previous" currentValue="Qz1JTg==">Qz1JTg==</fsN
 ame-shadow>'
...

I need to remove content starting from *-shadow to -shadow' for all instances.
i.e., My expected output is

objectClass: FSConfig
objectClass: extensibleObject
fsCAIP: 1.1.1.1
fsName: C=IN

How do I achieve this with sed/awk/grep/ anything else. Your help in this regard is much appreciated!

2 
awk '/shadow:/{x=1}/shadow>./{x=0;next}!x' yourfile

---------- Post updated at 03:44 PM ---------- Previous update was at 03:27 PM ----------

awk '{x=1}/shadow:/,/shadow>./{x=0}x' yourfile

---------- Post updated at 03:46 PM ---------- Previous update was at 03:44 PM ----------

awk '/shadow:/,/shadow>./{next}1' yourfile
1 Like
3

Thanks ctsgnb, works like a charm!

4

Hi,

There is one problem with the code you had provided.

dn: fsDomain=default
 fsDomain: default
 objectClass: FSDomainCredentials
 objectClass: extensibleObject
 fsAttribA: xyz
 fsAttribA-shadow: '<fsAttribA-shadow status="previous" currentValu
  e="MTk2ODU=">MTk2ODU=</fsAttribA-shadow>'
 fsAttribA-shadow: '<fsAttribA-shadow status="previous" currentValue="WmZ
  MdC11U3BZLURGVnYtZEpMeQ==">WmZMdC11U3BZLURGVnYtZEpMeQ==</fsAttribA-shadow>
  '
 fsAttribB: 19685

When I executed the command on the above LDAP content, fsAttribB gets lost.

dn: fsDomain=default
 fsDomain: default
 objectClass: FSDomainCredentials
 objectClass: extensibleObject
 fsAttribA: xyz

How can I overcome this?

5

Thanks,
But still some issue :frowning:
Now I get duplicate entries in my output.
Eg:
For input

fsAttribA-shadow: '<fsAttribA-shadow status="previous" currentValu
 e="MTk2ODU=">MTk2ODU=</fsAttribA-shadow>'
fsAttribA-shadow: '<fsAttribA-shadow status="previous" currentValue="WmZ
 MdC11U3BZLURGVnYtZEpMeQ==">WmZMdC11U3BZLURGVnYtZEpMeQ==</fsAttribA-shadow>
 '
fsAttribA: 19685

I get

fsAttribA: 19685
...
fsAttribA: 19685
6

  1. Please upload your original input file.

  2. Your initial requirement was to filter out things relating to "-shadow" stuff, not to look for duplicate entries.

If you don't care about the order, then you can just

awk '.... ' yourfile | sort | uniq

awk '.... ' yourfile | sort -u

Otherwise, please give all your requirements at once

---------- Post updated at 05:43 PM ---------- Previous update was at 05:38 PM ----------

I don't understand why you got duplicate entries : for me it worked fine :

$ cat f1
fsAttribA-shadow: '<fsAttribA-shadow status="previous" currentValu
 e="MTk2ODU=">MTk2ODU=</fsAttribA-shadow>'
fsAttribA-shadow: '<fsAttribA-shadow status="previous" currentValue="WmZ
 MdC11U3BZLURGVnYtZEpMeQ==">WmZMdC11U3BZLURGVnYtZEpMeQ==</fsAttribA-shadow>
 '
fsAttribA: 19685

$ awk '/shadow:/,/shadow>/{next}/^ *.$/{next}1' f1
fsAttribA: 19685
7

Hi,
input file and expected output files attached.
Thanks again!

8

Did you copy/paste into the old.txt file ?

How has the old.txt file been generated ?

9

please open with wordpad, notepad removes line formation.
old.txt is the output of the ldapsearch...

10 
awk 'NR<2{printf $0;next}{printf "%s",(/^ /?z:RS) $0}END{print RS}' yourfile | awk '/shadow:/,/shadow>/{next}/^ *.$/{next}1'

---------- Post updated at 06:47 PM ---------- Previous update was at 06:44 PM ----------

Yup, i know, what i meant was :

is old.txt the result of an unix command or script or whatever ?

or the result of a job that run in an application ? or from anywhere else ?

How has this input file been generated ?

Because it seems this file has been generated in a very bad way, with truncated lines even without a fixed length ... so it seems to be a very crap format that make me suspect that is has not been generated in the correct way it shoud have been.

---------- Post updated at 07:05 PM ---------- Previous update was at 06:47 PM ----------

To keep the fsCert and fsKey in 2 separate line and having their second line starting with a space, you can give a try to :

awk 'NR<2{printf $0;next}{printf "%s",(/^ /?z:RS) $0}END{print RS}' yourfile | awk '/shadow:/,/shadow>/{next}/^ *.$/{next}/^fs(Cert|Key)/{x=$NF;NF--;print $0 RS FS x;next}1'