# Dynamic Array Issue

**URL:** <https://community.unix.com/t/dynamic-array-issue/169696>\
**Category:** Shell Programming and Scripting\
**Created:** [March 11, 2007, 1:48pm UTC](https://community.unix.com/t/dynamic-array-issue/169696 "2007-03-11T13:48:37Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![ddedic](https://community.unix.com/letter_avatar/ddedic/32/6_449c533f586939b6e5f246632a46421f.png) [@ddedic](https://community.unix.com/u/ddedic)\
**Post date:** [March 11, 2007, 1:48pm UTC](https://community.unix.com/t/dynamic-array-issue/169696/1 "2007-03-11T13:48:37Z")

</div>

Could one of you, please, provide some input regarding my problem below and it is as follows:

I have 2 files that I need to make sure are identical before processing:

First, I sort both files

Second, I do a diff file1 file2 \> File 3  
This provides me with the difference.

Now, I need to read File 3 - into an array and remove the difference from both files - if any

filename = /.../.../.../... #input file  
set -A \<array\_name\> [\< $filename]

Could I use the sed '/expression/' to remove the line???

Please advise.

Don

---

<div class="post-metadata">

**Author:** ![Perderabo](https://community.unix.com/user_avatar/community.unix.com/perderabo/32/27_2.png) [@Perderabo](https://community.unix.com/u/Perderabo)\
**Post date:** [March 11, 2007, 2:39pm UTC](https://community.unix.com/t/dynamic-array-issue/169696/2 "2007-03-11T14:39:18Z")

</div>

> [@ddedic](#):
>
> I have 2 files that I need to make sure are identical before processing:

Just:  
cp file1 file2  
or copy the larger one to the smaller one (or vice versa).

---

<div class="post-metadata">

**Author:** ![ddedic](https://community.unix.com/letter_avatar/ddedic/32/6_449c533f586939b6e5f246632a46421f.png) [@ddedic](https://community.unix.com/u/ddedic)\
**Post date:** [March 11, 2007, 4:44pm UTC](https://community.unix.com/t/dynamic-array-issue/169696/3 "2007-03-11T16:44:28Z")

</div>

sorry, for not being more specific:

I am processing 2 large files - 10mil records

One is master and the other is detail. Because there is no uniqueness between them I have to rely on the row count of each file before processing.

With that said, I cut -d "|" -f2 from both files into \> aa.out and ab.out

Once this is complete, I sort both files and do a diff to get the records in one and not the other and vice versa.

Now that I have a diff.out (most of the time it will only be a record or two)  
I need to be able to remove them from the original file before application processing.

I will attempt to illustrate and for your review:

File 1  
111111  
222222  
333333  
555555

File 2  
111111  
222222  
333333  
444444

cat diff.out  
03c4400  
444444  
03c5556  
555555

I now need to remove those two records from both files - please let me know if any ideas.

---

<div class="post-metadata">

**Author:** ![Perderabo](https://community.unix.com/user_avatar/community.unix.com/perderabo/32/27_2.png) [@Perderabo](https://community.unix.com/u/Perderabo)\
**Post date:** [March 11, 2007, 6:29pm UTC](https://community.unix.com/t/dynamic-array-issue/169696/4 "2007-03-11T18:29:46Z")

</div>

I assume you have something like file1.orig and file2.orig with more fields but using | as a field separator. Here is a way to do this with comm and sed:

```nohighlight
$
$ cat file2.orig
111111|sdjksd
222222|sdjksd
333333|sdjksd
444444|sdjksd
$ comm -13 file1 file2 | sed 's=^=/^=;s=$=\|/d=' > file2.sed
$ sed -f file2.sed < file2.orig
111111|sdjksd
222222|sdjksd
333333|sdjksd
$

```

---

<div class="post-metadata">

**Author:** ![ddedic](https://community.unix.com/letter_avatar/ddedic/32/6_449c533f586939b6e5f246632a46421f.png) [@ddedic](https://community.unix.com/u/ddedic)\
**Post date:** [March 11, 2007, 10:38pm UTC](https://community.unix.com/t/dynamic-array-issue/169696/5 "2007-03-11T22:38:03Z")

</div>

thank you for your prompt response. This is exactly what I was looking for. Thank you.

---

<div class="post-metadata">

**Author:** ![ddedic](https://community.unix.com/letter_avatar/ddedic/32/6_449c533f586939b6e5f246632a46421f.png) [@ddedic](https://community.unix.com/u/ddedic)\
**Post date:** [March 11, 2007, 11:08pm UTC](https://community.unix.com/t/dynamic-array-issue/169696/6 "2007-03-11T23:08:06Z")

</div>

correct - i am using a "|" delimited file and the only common field between the 2 files is -f3 that i cut before comparing - the rest of the fields are different.

Specifically, I am struggling with step to remove those records from the individual files once difference has been determined.

Please let me know if you can help.

Example.

File 1 - master  
ab|dc|11111|2|3|4|5|6|  
ab|dc|22222|2|3|4|5|6|  
ab|dc|33333|2|3|4|5|6|  
ab|dc|55555|2|3|4|5|6|

File 2 - detail  
xa|zd|qs|11111|w|3|4|66|b  
xa|zd|qs|22222|w|3|4|8|d  
xa|zd|qs|33333|w|3|4|3|s  
xa|zd|qs|44444|w|3|4|1|f

===============================

After compare 😕 result should be:  
File 1 - master  
ab|dc|11111|2|3|4|5|6|  
ab|dc|22222|2|3|4|5|6|  
ab|dc|33333|2|3|4|5|6|

File 2 - detail  
xa|zd|qs|11111|w|3|4|66|b  
xa|zd|qs|22222|w|3|4|8|d  
xa|zd|qs|33333|w|3|4|3|s

---

<div class="post-metadata">

**Author:** ![Perderabo](https://community.unix.com/user_avatar/community.unix.com/perderabo/32/27_2.png) [@Perderabo](https://community.unix.com/u/Perderabo)\
**Post date:** [March 11, 2007, 11:56pm UTC](https://community.unix.com/t/dynamic-array-issue/169696/7 "2007-03-11T23:56:27Z")

</div>

```nohighlight
$ cat file2.orig
aa|bb|111111|sdjksd
aa|bb|222222|sdjksd
aa|bb|333333|sdjksd
aa|bb|444444|sdjksd
$ comm -13 file1 file2 | sed 's=^=/^[^|]*|[^|]*|=;s=$=\|/d=' > file2.sed
$ sed -f file2.sed < file2.orig
aa|bb|111111|sdjksd
aa|bb|222222|sdjksd
aa|bb|333333|sdjksd
$

```
