Remove duplicate lines which has been repeated 4 times

Remove duplicate lines which has been repeated 4 times attached test.txt

below command tried and not getting expect output.

for i in `cat test.txt | uniq`
        do
        num=`cat test.txt | grep $i | wc -l`
        echo $i $num
        done

test.txt

/usr/bin/gnome-keyring-daemon --daemonize --login
nome-session
/bus-launch --sh-syntax --exit-with-session
bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
/usr/libexec/gconfd-2
/usr/libexec/gnome-settings-daemon
eahorse-daemon
/usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
usr/libexec/gvfsd
metacity
/usr/bin/pulseaudio --start --log-target=syslog
gnome-panel
/usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
/usr/libexec/pulse/gconf-helper
/usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
/usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
nautilus
/usr/libexec/bonobo-activation-server --ac-activate --ior-output-fd=18
/usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
/usr/libexec/gvfs-gdu-volume-monitor
/usr/libexec/wnck-applet --oaf-activate-iid=OAFIID:GNOME_Wncklet_Factory --oaf-ior-fd=18
/usr/libexec/trashapplet --oaf-activate-iid=OAFIID:GNOME_Panel_TrashApplet_Factory --oaf-ior-fd=24
/usr/lib/vmware-tools/sbin32/vmtoolsd -n vmusr --blockFd 3
usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
gpk-update-icon
gnome-power-manager
gnome-volume-control-applet
bluetooth-applet
/usr/sbin/restorecond -u
usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
/usr/libexec/polkit-gnome-authentication-agent-1
python /usr/share/system-config-printer/applet.py
/usr/libexec/gdu-notification-daemon
  usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
nm-applet --sm-disable
usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
/usr/libexec/gvfs-afc-volume-monitor
/usr/libexec/gvfsd-trash --spawner :1.7 /org/gtk/gvfs/exec_spaw/0
/usr/libexec/gvfs-gphoto2-volume-monitor
gnome-screensaver
/usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
/usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
/usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
/usr/libexec/clock-applet --oaf-activate-iid=OAFIID:GNOME_ClockApplet_Factory --oaf-ior-fd=29
/usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
/usr/libexec/notification-area-applet --oaf-activate-iid=OAFIID:GNOME_NotificationAreaApplet_Factory --oaf-ior-fd=39
/usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
/usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
/usr/libexec/gvfsd-metadata
/usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
/usr/bin/gnome-terminal -x /bin/sh -c cd '/home/pintu/Desktop' && exec $SHELL -l
/usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
gnome-pty-helper
/usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
/bin/bash -l

If I understand you correctly, you want to keep only the first four unique lines?

awk '++A[$0] < 5' file

If u see the test.txt below lines are repeated 4 times wanted remove those line in to test.txt

/usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
/usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23

awk 'A[$0]++ > 4 ' test.txt

Aha, I see. Well that takes a little bit more work. Stand by! And you're welcome :rolleyes:

So - lines repeated up to three times should be printed as are - once, twice, or thrice. Then, lines repeated four or more times should be printed once?

Thanks
lines repeated up to 4 times should be printed as once

And what about lines appearing more than four times?

We can remove or display the line which has been repeated 4 times

Make up your mind! You want to display the line or remove it? You can't do both.

To display any line repeated at least (read as exactly) four times only once:

awk '++A[$0] == 4' file

Hi.

This is our fault: we should have required the OP to post the expected / desired output from the sample input.

I interpreted ( and interpolated ) and tested at least twice ... cheers, drl

Thanks quick respond

Missed first coulmn in input file have attaced input file
The output should display
lines has been deleted repeated 4 times

lines has been deleted repeated 4 times
345 /usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
021 /usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
2341 /usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
231 /usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
222 /usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
221 /usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
221 /usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
1213 /usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
12 usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
22 usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
44 usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
2367 usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35

Then, instead of the whole line, compare
only column $2

awk 's[$2]++<4' test.txt

or
the whole line( $0 ) but column $1

awk '{x=$0; $1=""} s[$0]++<4 {print x}' test.txt

<4 means up to 4 instances are shown.

Thanks a lot for ur replay
not getting correct output
While using below command getting wrong output

 awk '{x=$0; $1=""} s[$0]++<4 {print x}' test.txt
 

 /usr/bin/gnome-keyring-daemon --daemonize --login
nome-session
/bus-launch --sh-syntax --exit-with-session
bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
/usr/libexec/gconfd-2
/usr/libexec/gnome-settings-daemon
eahorse-daemon
/usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
/usr/bin/pulseaudio --start --log-target=syslog
/usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
/usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
/usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
/usr/libexec/bonobo-activation-server --ac-activate --ior-output-fd=18
/usr/libexec/wnck-applet --oaf-activate-iid=OAFIID:GNOME_Wncklet_Factory --oaf-ior-fd=18
/usr/libexec/trashapplet --oaf-activate-iid=OAFIID:GNOME_Panel_TrashApplet_Factory --oaf-ior-fd=24
/usr/lib/vmware-tools/sbin32/vmtoolsd -n vmusr --blockFd 3
usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
/usr/sbin/restorecond -u
usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
python /usr/share/system-config-printer/applet.py
  usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
nm-applet --sm-disable
usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
/usr/libexec/gvfsd-trash --spawner :1.7 /org/gtk/gvfs/exec_spaw/0
/usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
/usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
/usr/libexec/clock-applet --oaf-activate-iid=OAFIID:GNOME_ClockApplet_Factory --oaf-ior-fd=29
/usr/libexec/notification-area-applet --oaf-activate-iid=OAFIID:GNOME_NotificationAreaApplet_Factory --oaf-ior-fd=39
/usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
/usr/bin/gnome-terminal -x /bin/sh -c cd '/home/pintu/Desktop' && exec $SHELL -l
/usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
 

it shoud be display the line repeated 4 times

 345 /usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
021 /usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
2341 /usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
231 /usr/libexec/gvfsd-burn --spawner :1.7 /org/gtk/gvfs/exec_spaw/1
222 /usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
221 /usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
221 /usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
1213 /usr/bin/gnote --panel-applet --oaf-activate-iid=OAFIID:GnoteApplet_Factory --oaf-ior-fd=23
12 usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
22 usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
44 usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35
2367 usr/libexec/gdm-user-switch-applet --oaf-activate-iid=OAFIID:GNOME_FastUserSwitchApplet_Factory --oaf-ior-fd=35

 

---------- Post updated at 11:06 PM ---------- Previous update was at 02:01 PM ----------

Thanks to all and MadeInGermany

for ur quick respond

Issue resolved

This produces the output from your last post, but the first 4 occurrences rather than random 4 occurrences

awk '{p=$1; $1=""; pid[$0,++s[$0]]=p} s[$0]==4 {for (i=1; i<=s[$0]; i++) print pid[$0,i] $0}' test.txt

1-Getting bug It is printing exact first 4 occurrences but if the lines are repeated more then 4 those line are not printing,need those line also if possible

2-few step am not able to understand if possible could please example with comments

ps -u gpoper -o pid= -o args=| awk '{p=$1; $1=""; pid[$0,++s[$0]]=p} s[$0]==4 {for (i=1; i<=s[$0]; i++) print pid[$0,i] $0}' 

---------- Post updated at 08:39 PM ---------- Previous update was at 08:37 PM ----------

*explain

Print the previous (stored) lines if s[$0] == 4 and the current line for s[$0] > 4

awk '{p=$1; $1=""; pid[$0,++s[$0]]=p} s[$0]==4 {for (i=1; i<=s[$0]; i++) print pid[$0,i] $0} s[$0]>4 {print p,$0}' test.txt

Or print the repeated lines next to each other in the END section (after the last line was read).

awk '{p=$1; $1=""; pid[$0,++s[$0]]=p} END {for (is in s) if (s[is]>=4) for (i=1; i<=s[is]; i++) {print pid[is,i],is}}' test.txt

Thanks a lot its exactly matching my requirement .

---------- Post updated 12-07-17 at 12:30 AM ---------- Previous update was 12-06-17 at 09:23 PM ----------

Please share if some AWK doc u have

I usually consult the man pages

man awk
...

man sed
...

Not made for the beginner, but one can get used to it.