how do I filter double lines from a txt file

Shell Programming and Scripting
1

Hi,

I have a file with the following:

access-list.txt

router1
access-list 1 permit any any
access-list 1 deny any any
router2
access-list 2 permit any any
access-list 2 deny any any
router3
access-list 3 permit any any
access-list 3 deny any any

I want to hava an output that I only see 1 line for the access-lists.
So the output has to be:

router1
access-list 1 permit any any
router2
access-list 2 permit any any
router3
access-list 3 permit any any

How do I do this?

2 
awk ' { print ; if( match( $1 , "access-list" ) ) { getline } } ' file
3 
awk '{ if( NF > 3) { if ( $0 ~ /permit/ ) { print } } else { print } }' filename
4

Hi,

Thanks for the 2 commands you provided.
But it's not working well ...

I have another file see below ...

So what I want is that it only shows 1 access-list line if the number is allready there.

I have a list 81155 access-list lines with 2370 different routers in this file

the output of the file looks like this:

=====================
NEXT HOST
=====================
 AEADBAS001
ip access-list extended BLA_Incoming_Filter
ip access-list extended BLA_Outgoing_Filter
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 deny   any log
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 deny   xxxxxxxxxxxxxx
=====================
NEXT HOST
=====================
 AEADBMP002
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 deny   any log
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 deny   any log
access-list 22 permit xxxxxxxxxxxxxx
access-list 22 permit xxxxxxxxxxxxxx
=====================
NEXT HOST
=====================
 AEADBMP003
ip access-list extended GAA_Incoming_Filter
ip access-list extended GAA_GW_Outgoing_Filter
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 deny   any log
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 2 deny   any log

I only want to filter out the double lines

so the output need to be:

=====================
NEXT HOST
=====================
 AEADBAS001
ip access-list extended BLA_Incoming_Filter
ip access-list extended BLA_Outgoing_Filter
access-list 1 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx

=====================
NEXT HOST
=====================
 AEADBMP002
access-list 1 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx
access-list 22 permit xxxxxxxxxxxxxx
=====================
NEXT HOST
=====================
 AEADBMP003
ip access-list extended GAA_Incoming_Filter
ip access-list extended GAA_GW_Outgoing_Filter
access-list 1 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx

can you help me with that command?

Thanks

5 
awk ' { if(match( $0 , "^access-list" ) ){ if( x != $2 ){ x=$2;print } } else print } ' file
6

Thanks anbu23,

This works great!

I think I need to start learning this AWK scripting also ...

Thanks,

7

anbu23

Can you please explain me this line :

What I understand of the first part is:
if something is found that begins (^) with access-list trough the complete document ($0) ... then ???

I have no idea what this means :
{ if( x != $2 ){ x=$2;print } } else print }

Can you please explain how it works ...

8 
access-list 1 permit xxxxxxxxxxxxxx
access-list 1 permit xxxxxxxxxxxxxx
access-list 2 permit xxxxxxxxxxxxxx

In above sample when it reads the first line x != $2 is satisfied and 1 is assigned to x and this line is printed. In case of second line x != $2 is not satisfied and the line is not printed. In case of third line x != $2 is satisfied and 2 is assigned to x and this line is printed.

9

I still dont understand it

10

Can you tell what do you understand now?

11

First it picks up the records begining with "access-list"

after that (Note: $2 represents the second field( that is 1 for the first line))
the statement x!=$2 would be true for the first time. and then 1 is being assigned
to x so x has the value 1. when the command processes the second line then $2=1 that
means x=$2 so this time the condition fails and so nothing is printed uptill it reaches
where $2 value is 2 it would continue to skip the lines. once it has reached where $2=2
then again the condition(x!=$2) is true and the record is printed and this time 2 is
assigned to x and this continues uptill the end of file. Incase, if the record doesnt
start with "access-list" then also print the line.

I hope i am clear enough

ANBU - If i am wrong please correct me.