Compare two files line by line

Shell Programming and Scripting
1

Query:
There are two files as below -
/home/rgupta/input/file.txt.arch (source file)
/home/rgupta/output/file.txt (destination file)

Files details are attached for the reference.

Scenario:
File /home/rgupta/input/file.txt.arch picked up by an application xyz. Application xyz does some validation/checks on this file then finally kept at /home/rgupta/output/file.txt.
While copying the file at /home/rgupta/output, Following activities can happen by application xyz -
1) some records can be eliminated
2) There can be some records modified

There are many files with huge number of records with comma/semicolon separated fields.
I need to write a shell script which will compare source file with destination file line by line. And this script should create another file with following data -

  1. If record is missing in destination file -
    Source file name, destination file name, missing record, missing record number in source
    file
  2. If record is modified while transferring the file from source to destination -
    • Source file name, Data record from source file [first line in new file]
    • Destination file name, Data record from Destination file [second line in new file]
    • modified record number, exact field which is modified [second line in new file]

I tried following command -
sdiff -s /home/rgupta/input/file.txt.arch /home/rgupta/input/file.txt
Output is -
6015a6016
> 29429293,1,,,01387262543,N,01387262543,N,0,01387262543,N
10000c10001
COUNT=9999 | COUNT=10000;

Output of this command is similar what I expect but the missing record is coming incomplete.

Could someone please provide me the solution?

file1.txt:

file1.txt
029429284,1,,,02077232651,N,02077232651,N,0,02077232651,N,0,0,1877,02072496953,U,N,,02072496953,U,20100726111641,20100726111641,0,20100726111714,00000330,16,16,10,N;
029429285,1,,,02087488096,N,02087488096,N,0,02087488096,N,0,0,1877,02083325000,U,N,,02083325000,U,20100726111611,20100726111611,0,20100726111715,00001040,16,16,10,N;
029429286,1,,,02072432251,N,02072432251,N,0,02072432251,N,0,0,1363,02076241470,U,N,,02076241470,U,20100726111257,20100726111257,0,20100726111701,00004040,16,16,10,N;
029429287,1,,,02087447700,N,02087447700,N,0,02087447700,N,0,0,1363,07826378298,U,N,,07826378298,U,20100726111650,20100726111650,0,20100726111701,00000110,16,16,10,N;
029429288,1,,,02087400376,N,02087400376,N,0,02087400376,N,0,0,8010,08000151736,U,N,,08000151736,U,20100726111237,20100726111237,0,20100726111651,00004140,16,16,10,N;
029429289,1,,,02088736705,N,02088736705,N,0,02088736705,N,0,0,6160,003531437509,U,N,,003531437509,U,20100726111644,20100726111644,0,20100726111651,00000070,16,16,10,N;
029429290,1,,,02083995696,N,02083995696,N,0,02083995696,N,0,0,1363,07803049166,U,N,,07803049166,U,20100726111700,20100726111700,0,20100726111704,00000040,16,16,10,N;
029429291,1,,,01228631982,N,01228631982,N,0,01228631982,N,0,0,1363,07866387607,U,N,,07866387607,U,20100726121123,20100726121123,0,20100726122312,00011490,16,16,10,N;
029429292,1,,,01620844940,N,01620844940,N,0,01620844940,N,0,0,1877,01312402295,U,N,,01312402295,U,20100726121700,20100726121700,0,20100726122251,00005510,16,16,10,N;
029429294,1,,,01463783624,N,01463783624,N,0,01463783624,N,0,0,1363,01463811679,U,N,,01463811679,U,20100726122255,20100726122255,0,20100726122334,00000390,16,16,10,N;
029429295,1,,,01315537520,N,01315537520,N,0,01315537520,N,0,0,1363,08449999999,U,N,,08449999999,U,20100726121407,20100726121407,0,20100726122315,00009080,16,16,10,N;
029429296,1,,,01624844675,N,01624844675,N,0,01624844675,N,0,0,6160,02890614024,U,N,,02890614024,U,20100726121837,20100726121837,0,20100726122235,00003580,16,16,10,N;
029429297,1,,,01539723145,N,01539723145,N,0,01539723145,N,0,0,1363,09041619999,U,N,,09041619999,U,20100726122221,20100726122221,0,20100726122300,00000390,16,16,10,N;
029429298,1,,,01361884551,N,01361884551,N,0,01361884551,N,0,0,1363,01890818065,U,N,,01890818065,U,20100726121903,20100726121903,0,20100726122244,00003410,16,16,10,N;
029429299,1,,,02089944041,N,02089944041,N,0,COUNT=9999

file1.txt.arch:

029429284,1,,,02077232651,N,02077232651,N,0,02077232651,N,0,0,1877,02072496953,U,N,,02072496953,U,20100726111641,20100726111641,0,20100726111714,00000330,16,16,10,N;
029429285,1,,,02087488096,N,02087488096,N,0,02087488096,N,0,0,1877,02083325000,U,N,,02083325000,U,20100726111611,20100726111611,0,20100726111715,00001040,16,16,10,N;
029429286,1,,,02072432251,N,02072432251,N,0,02072432251,N,0,0,1363,02076241470,U,N,,02076241470,U,20100726111257,20100726111257,0,20100726111701,00004040,16,16,10,N;
029429287,1,,,02087447700,N,02087447700,N,0,02087447700,N,0,0,1363,07826378298,U,N,,07826378298,U,20100726111650,20100726111650,0,20100726111701,00000110,16,16,10,N;
029429288,1,,,02087400376,N,02087400376,N,0,02087400376,N,0,0,8010,08000151736,U,N,,08000151736,U,20100726111237,20100726111237,0,20100726111651,00004140,16,16,10,N;
029429289,1,,,02088736705,N,02088736705,N,0,02088736705,N,0,0,6160,003531437509,U,N,,003531437509,U,20100726111644,20100726111644,0,20100726111651,00000070,16,16,10,N;
029429290,1,,,02083995696,N,02083995696,N,0,02083995696,N,0,0,1363,07803049166,U,N,,07803049166,U,20100726111700,20100726111700,0,20100726111704,00000040,16,16,10,N;
029429291,1,,,01228631982,N,01228631982,N,0,01228631982,N,0,0,1363,07866387607,U,N,,07866387607,U,20100726121123,20100726121123,0,20100726122312,00011490,16,16,10,N;
029429292,1,,,01620844940,N,01620844940,N,0,01620844940,N,0,0,1877,01312402295,U,N,,01312402295,U,20100726121700,20100726121700,0,20100726122251,00005510,16,16,10,N;
029429293,1,,,01387262543,N,01387262543,N,0,01387262543,N,0,0,8091,01877242558,U,N,,01877242558,U,20100726122241,20100726122241,0,20100726122257,00000160,16,16,10,N;
029429294,1,,,01463783624,N,01463783624,N,0,01463783624,N,0,0,1363,01463811679,U,N,,01463811679,U,20100726122255,20100726122255,0,20100726122334,00000390,16,16,10,N;
029429295,1,,,01315537520,N,01315537520,N,0,01315537520,N,0,0,1363,08449999999,U,N,,08449999999,U,20100726121407,20100726121407,0,20100726122315,00009080,16,16,10,N;
029429296,1,,,01624844675,N,01624844675,N,0,01624844675,N,0,0,6160,02890614024,U,N,,02890614024,U,20100726121837,20100726121837,0,20100726122235,00003580,16,16,10,N;
029429297,1,,,01539723145,N,01539723145,N,0,01539723145,N,0,0,1363,09041619999,U,N,,09041619999,U,20100726122221,20100726122221,0,20100726122300,00000390,16,16,10,N;
029429298,1,,,01361884551,N,01361884551,N,0,01361884551,N,0,0,1363,01890818065,U,N,,01890818065,U,20100726121903,20100726121903,0,20100726122244,00003410,16,16,10,N;
029429299,1,,,02089944041,N,02089944041,N,0,02089944041,N,0,0,1877,02085601193,U,N,,02085601193,U,20100726111138,20100726111138,0,20100726112927,00017490,16,16,10,N;
COUNT=1000;

Highlighted record is extra record in source file.

2

Hi.

       -w NUM  --width=NUM
              Output at most NUM (default 130) columns per line.  

-- excerpt from man sdiff

for the sdiff on my system:

sdiff (GNU diffutils) 2.8.1

so that by default only the first 60 or so characters from each line of a differing set will be displayed.

Expanding the line width with sdiff might make the output even more unreadable, so if you need the entire line(s) displayed, perhaps the regular diff would be better ... cheers, drl

3

Please help me to write an shell script to compare above two files in a way so that three files can be generated with following data -

1) First file should have the records which are missing in file1.txt but available in file1.txt.arch (Red colored record).

2) Second file should have records which have been modified (Blue colored).

3) Third file should have the common records.

Please help me to provide the solution on this.

Thanks very much in advance.

Ravi