Extract delta records using with "comm" and "sort" commands combination

UNIX for Beginners Q & A
1

Hi All,
I have 2 pipe delimited files viz., file_old and file_new . I'm trying to compare these 2 files, and extract all the different rows between them into a new_file .

comm -3 < sort file_old < sort file_new > new_file

I am getting the below error:

-ksh: sort: cannot open [No such file or directory]

But if I do as below, it is successfully giving me the difference.

sort file_old > file1
sort file_new > file2
comm -3 file1 file2 > file3

Can you please help me understand why the first approach resulting in an error?

Thank you.

PS: I did search the forum before posting this thread but no luck

2

I think you need to deploy ksh 's "process substitution":

comm -3 <(sort file_old) <(sort file_new) > new_file
3

It displayed syntax error if I include brackets "()".

comm -3 < (sort file1) < (sort file3) > file3
-ksh: syntax error: `(' unexpected

comm -3 < (sort file1) < (sort file3) > (file3)
-ksh: syntax error: `(' unexpected
4

That's NOT what I posted!

5

To expand a little bit on what RudiC already said...

If you're trying to perform process substitution, there can't be any space between the redirection operator and the (utility [argument...]) .

To avoid confusion in the future, note that ( and ) are called parentheses; brackets are [ and ] and braces are { and } . Brackets, braces, and parentheses all have distinct meaning in the shell command language; they are not interchangeable.

Note also that process substitution is a fairly new feature in ksh and is not available in all versions of the Korn shell. And, you haven't told us which version of ksh you're using. It was not available in any version of ksh before 1993 and was not in the early version of ksh93 . It is in version 93u+ from August 1, 2012, but I am not sure which version first included process substitution.

6

Thank you Don for explaining in so much detail. Please see below, getting same error even after removing the spaces between "<"/">" and following command.

$ ksh --v
  version         sh (AT&T Research) 93u+ 2012-08-01
$ comm -3 <sort file_old <sort file_new >new_file
-ksh: sort: cannot open [No such file or directory]

---------- Post updated at 04:19 PM ---------- Previous update was at 04:16 PM ----------

It didn't work with parenthesis also.

$ comm -3 <(sort file_old) <(sort file_new) >new_file
-ksh: sort: cannot open [No such file or directory]
7

Are you pointing to the correct directory?
Does that directory exist?
Are 'file_old' and 'file_new' in the current directory?
Do both in fact exist?

8

Apologies for the delayed response. Didn't realize wisecracker responded.

Yes wisecracker, when I first posted my query, all the files were in the current directory. The 3 steps process of first sorting the 2 files and saving into other files, followed by applying "comm" command on the sorted files succeeded.

I tried all the commands again, this time the "comm" also worked. I was puzzled how it worked this time.

My guess is when I first posted my query, as RudiC pointed I didn't apply "process substitution"; later when I did, I guess someone else removed those files from my current directory during that time that I didn't realize.

Please see below:

$ ls -ltr
total 134808
-rw-r--r-- 1 xbbldyx y8dxmr     14008 Aug 16 23:36 file_old
-rw-r--r-- 1 xbbldyx y8dxmr     15033 Aug 16 23:37 file_new
..
..
$ comm -3 < sort file_old < sort file_new > new_file
-ksh: sort: cannot open [No such file or directory]
$ comm -3 <sort file_old <sort file_new > new_file
-ksh: sort: cannot open [No such file or directory]
$ comm -3 <(sort file_old) <(sort file_new) > new_file
$ more new_file
        ||G6||||Y||||1000|5000|6/4/2008|||3||6/16/2166|1010|xyz|Y||||FR|FR|FR|0.014|JR|GEN||||N|FR||8.0|USD||1
$ diff file_old file_new
10a11
>||G6||||Y||||1000|5000|6/4/2008|||3||6/16/2166|1010|xyz|Y||||FR|FR|FR|0.014|JR|GEN||||N|FR||8.0|USD||1

$ sort file_old > file1
$ sort file_new > file2
$ comm -3 file1 file2 > file3
$ diff new_file file3
$

Thank you RudiC for the tip, Don for clarification and wisecracker for prompting me to check if the files exist. Thank you all.