Multiple file handling

UNIX for Advanced & Expert Users
1

Dear All,

I have two files, which looks like:

File 1

124
235
152
178
156
142
178
163
159

File 2
124|5623
452|6698
178|9995
235|7542
159|8852
152|9963
156|8512
885|9956
754|6523

I have to search all values of File1 in File2 and make an output file which should look like:

124|5623
235|7542
152|9963
178|9995
156|8512
142|
178|9995
163|
159|8852

The sequence of output file Should be in accordance to File 1. If any Value of File 1 is not found in File 2 then second field
of Output file should be a blank.
I cannot use grep command in for loop as the number of entries in File 1 is about 5000 and that in File 2 is 900,000.

Regards
Rochit

2

just one of many ways:

awk 'BEGIN {  
        while ( (getline line < "file2")  > 0 ) {
              n = split(line,a,"|")
              array[a[1]] = a[2]
        }
    } 
    {  print $1 "|" array[$1]  }
' "file1"

output:

# ./test.sh
124|5623
235|7542
152|9963
178|9995
156|8512
142|
178|9995
163|
159|8852
3

Thanks but it is taking lot of time to execute and I am afraid I cannot devote that much of time. So If u could please suggest another solution which gives an output faster than this one.
And Please could u explain how the code u have given works.

Thanks & regards
Rochit

4

my bad. i missed the 900000 records. well how about trying something else. if you have Python, this is an alternative:

data=open("file1").readlines() #read file1 into array
data=[i.strip() for i in data] #get rid of newlines
for line in open("file2"):     #process file2 line by line
    if line[0:3] in data: #check only first 3 characters
        print line,
    else: print "%s|" % line[0:3]
5

hi,
sorry to say but I think ,y server does has Python installed in it. I can execute awk and perl scripts.
Thanks & regards
Rochit

6

sorry, i don't code perl, but roughly the method is same.
not tested code:

open(file1,"<","file1") or die "cannot open file1:$!";
#get file1 into array
while ( <file1> ) {
   push @file1, chomp($_); 
}
open(file2,"<","file2") or die "cannot open file2:$!";
#go through every line in file2
while ( my $line = <file2> ) {
  chomp($line);
  # get the first 3 characters, check if its in file1 array, if yes print
  if ( exists $file1[substr($line,0,3)] ) {
   print $line;
  }
}
close(file1);
close(file2);
7 
awk -F"|" ' NR == FNR { arr[$1]=$2; next } {  print $1"|"arr[$1] } ' file2 file1
8

Thanx. The code provided by you really worked. Can u please explain the code. I shall be very thankful.

9

quick one with perl !

#! /opt/third-party/bin/perl

open(FILE, "<", "second") || die "Unable to open file <$!>\n";

while(<FILE>) {
  chomp;
  @arr = split(/\|/);
  $fileHash{$arr[0]} = $arr[1];
}

close(FILE);

open(FILE, "<", "first") || die "Unable to open file <$!>\n";

while(<FILE>) {
  chomp;
  if( exists $fileHash{$_} ) {
    print "$_|$fileHash{$_}\n";
  }
  else {
    print "$_|\n";
  }
}

close(FILE);


exit 0