Delete unique rows - optimize script

Shell Programming and Scripting
1

Hi all,

I have the following input - the unique row key is 1st column

cat file.txt
 
[4] A response
[1] C request
[1] C response
[3] D request
[2] C request
[2] C response
[5] E request

The desired output should be

[1] C request
[1] C response
[2] C request
[2] C response

Now i have implemented the below loop which does work but when the input file is bigger than 300 mb in size the whole process of removing the non-pairs rows takes ages since it needs to scan the whole file in a loop.

#/bin/bash
req=$(mktemp)
res=$(mktemp)
new=$(mktemp)
tmp=$(mktemp)
grep request  $1 > $req
grep response $1 > $res
for id in `cat $req | awk '{ print $1}'` 
do    
    id=$(echo $id | tr -d "[]")
    grep "$id" $res > $tmp 
    if [[ -s $tmp ]]
    then
 grep "$id" $req >> $new
 cat $tmp >> $new 
    fi
done
mv $new $2
rm $req $res $tmp

Any idea how i can optimize/ do it differently to remove the unique rows as per above example in order to speed up the process?

2 
[4] A response
[1] C request
[1] C response
[3] D request
[2] C request
[2] C response
[5] E request

I think you want all the response items as per your request?????

#/bin/bash
req=$(mktemp)
res=$(mktemp)
new=$(mktemp)
tmp=$(mktemp)
grep request  $1 | sort | uniq > $req   #to get unique request items.
grep response $1 > $res
for id in "$(cat $req | awk '{ print $1,$2}')" # Get whole part as - [1] C,[2] C -- Makes it more unique
do    
    #id=$(echo $id | tr -d "[]")   # I think its better to search with [1] C to get more unique results.. that's why commented this.
    grep "$id" $res > $tmp 
    if [[ -s $tmp ]]
    then
 grep "$id" $req >> $new
 cat $tmp >> $new 
    fi
done
mv $new $2
rm $req $res $tmp
3

Thanks for quick reply.

I want only request/response pairs. IF there is only request or response without any associated pair then it should be removed.

Cheers.

4

Have you tried my suggestions..?

5

Yes i did - but there is no difference since is the same logic, for each row it needs to scan the whole file in a loop.

I think i'll gain more if i find a way to scan the file once and delete the unique rows/ lines, something like this....

check for key "[1] request" or "[1] response" and if you don't find the pair delete it..

6

Try this..

sort file | awk '/request/ {s=$0;p=$1;getline} {if ($0 ~ /response/ && $1 == p) { print s"\n"$0 }}'
7

One way :

~/unix.com$ awk 'NR==FNR{A[$1]++;next}A[$1]==2' file.txt file.txt
8

@pamu, @tukuyomi - both solutions works so many thanks!