awk or uniq

UNIX for Beginners Q & A UNIX for Dummies Questions & Answers
1

Hi Help,

I have a file which looks like

1 20 30 40 50 60 6
2 20 30 40 50 60 8
7 20 30 40 50 60 7
4 30 40 50 60 70 8
5 30 40 50 60 70 9
2 30 40 50 60 70 8

I want the o/p as

1 20 30 40 50 60 6
4 30 40 50 60 70 8

Is there a way I can use uniq command or awk to do this?

Thanks,

2

You cannot use the uniq command for this because uniq filters lines so that only unique ones (hence the name) remain. As all your lines are unique, because they all differ in some regards uniq will not give you what you want.

awk might be able to give you the result you want once you can tell us what exactly the rules for your intended filtering are. I might be slow but i do not get it from a few example lines and two lines of expected output. Depending on these rules you might also take sort into consideration as a probably helpful tool.

I hope this helps.

bakunin

3

Thanks Bakunin,

For, NR=1,2,3; $2,$3,$4,$5,$6

are exact same so I would grab the

NR==1

only.

For, NR=4,5,6; $2,$3,$4,$5,$6

are exact same so I would grab the

NR==4

only

It looks complicated.
Thanks,

4 
awk '!A[$2,$3,$4,$5,$6]++' file
5

Hi Yoda,
I am not sure why am I getting an error msg 'A[ event not found'.

6

Complicated is OK. Usually the complication goes away after correctly describing the problem. Just as a general suggestion: more than 30 years in IT have proven to me, over and over, that once a problem is correctly defined and stated the solution is in most cases obvious and a direct outcome of the problem definition.

Coming back from general aspects to your problem at hand: you want for every group of lines which have the same values in the second, thrid, fourth, fifth and sixth column only the first line to be printed and all others suppressed yes?

Question: is white space delimiting the columns to be considered, i.e are the lines

1 20 30 40 50 60 6
1 20  30 40 50 60 6

considered to be equal or not? Or can't that happen?

Yodas solution, for instance, may be working for you or not, depending on your answer to this question.

Are there leading/trailing whitespace in the lines? Need they to be taken into account or not? Will the white space always be formed by the same character or might there be space and tabs mixed? Are your values always numeric or might they be alphanumeric?

bakunin

7

Using sort:

sort -m -u -k 2,6
1 20 30 40 50 60 6
4 30 40 50 60 70 8
8

I guess the exclamation point in the program is causing the issue.

You can try this instead:-

awk 'A[$2,$3,$4,$5,$6]++==0' file
9

Hi mjf,
your script worked charm!
Thanks,

---------- Post updated at 05:08 PM ---------- Previous update was at 05:07 PM ----------

Many thanks Yoda!

10

This one works on adjacent like unique

awk '(k=$2 FS $3 FS $4 FS $5 FS $6)!=pk {print; pk=k}' file
11

Did you copy it exactly as presented? bash 's history functionality may get in the way:

!A[
bash: !A[: event not found

What's your shell?

12

These are sage words, and worth quoting.