How to sort common arrays from two different data files

I have two files with 2 columns each , together these two column make up a pair that I need. There are certain pairs that are common in both files. I need to create 3 files, one with the common pairs, and two files for exclusive pairs of each mother file.
It should be noted that common pairs may appear in any row of the other file.
I am using bash on ubuntu 18.04. I have tried with fortran, but for large files, it is crashing.
Please forgive me if I am unable to express my problem properly. Thank you in advance

File 1

 4   8
 4   76
 4   9
 4   73
 4   12
 4   69
 4   72
 5   9
 6   10
 7   11
 7   115
 8   12
 8   13
 8   65
 8   69
 8   76
 9   65
 9   69
 9   13
 10   14
 11   15
 11   39
 12   39
 12   16
 12   35
 12   68
 12   65
 12   69
 12   72
 12   76
 13   65
 13   69
 13   17
 14   18
 14   19
 15   19
 15   35
 15   38
 15   39
 15   68
 16   68
 16   27
 16   35
 16   20
 16   26
 16   65
 16   63
 16   25
 16   64
 18   38
                     

File 2

 4  8
 5  9
 6  10
 7  11
 8  12
 8  69
 9  65
 9  69
 9  13
 10  14
 11  15
 12  68
 12  65
 12  69
 12  72
 12  76
 12  73
 12  16
 13  65
 13  17
 14  18
 15  19
 15  68
 15  72
 16  68
 16  27
 16  65
 16  63
 16  25
 16  26
 16  20
 17  21
 18  35
 19  31
 19  35
 19  27
 19  32
 19  36
 19  68
 19  26
 19  28
 20  27
 20  26
 20  31
 20  24
 20  25
 21  31
 21  34
 21  35
 22  31

Hi @pkdas,

using Fortran nowadays is quite unusual Here is an approach with python3:

#!/usr/bin/python3

# convert to int in order to make tuples sortable by number, not by string
f1 = set(tuple(map(int, ln.split())) for ln in open('f1'))
f2 = set(tuple(map(int, ln.split())) for ln in open('f2'))
# sets by definition haven't any order, so sort them
print('\n'.join('{} {}'.format(*t) for t in sorted(f1 & f2)), file=open('f.both', 'w'))
print('\n'.join('{} {}'.format(*t) for t in sorted(f1 - f2)), file=open('f.only-1', 'w'))
print('\n'.join('{} {}'.format(*t) for t in sorted(f2 - f1)), file=open('f.only-2', 'w'))

And for verification in bash:

#!/bin/bash

for f in f.both f.only*; do
    echo "${f#*.}:"
    # read pairs from outfiles and search for them in both infiles
    while read x y; do
        # \b matches at the edge of a word/number
        # that's only because of the different spaces in the infiles,
        #   otherwise "^$x $y$" would be enough, and the sed wouln't be needed
        # then sort by 1st number & strip multiple spaces
        grep -E "\b$x\b.*\b$y\b" f1 f2 | sort -g -k2 | sed -r 's,[[:space:]]+, ,g'
    done < $f
done

There are of course other and faster methods, especially with awk, or (maybe) with tools like cmp, join, paste etc. I only suggested this because it's a typical example for sets in python.

# compare the files - output in three tab'd columns
# use awk to build your three output files. 
comm --output-delimiter=$'\t' <(sort file1) <(sort file2) | 
 awk -F"\t" '{if($1 != ""){print $0 >"same.txt"}};
                    {if($2 != ""){print $0 >"left.txt"}};
                    {if($3 != ""){print $0 >"right.txt"}}'

The same comm solution step by step (using less memory)
Are the input files sorted?
If not then sort them

sort -o file1 file1
sort -o file2 file2

Is the spacing consistent?
Assuming yes.
Then create the output files

comm -23 file1 file2 >only_file1
comm -13 file1 file2 >only_file2
comm -12 file1 file2 >common_file1_2

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.