Adding new line to file

Shell Programming and Scripting
1

Hi everyone,
currently I writing a script for comparing 2 variable in 2 line then output the line with equal value to new file. However, the new file only contain last line only, the earlier line was delete. I do google my problem but still not find the way out. Sorry for my English.
Thank you very much in advance :).

Here is my script somehow look like:

for(tmp1=1; tmp1<=cnt1; tmp1++){
$line1= `head -tmp1 file1| tail -1`;
@str1=split(/\s/,$line1);
       for(tmp2=1; tmp2<=cnt2; tmp2++){
           $line2= `head -tmp2 file2| tail -1`;
           @str2=split(/\s/,$line2);
           open (OUT, ">out");
           if ($str1[3] eq $str2[3]){
              print OUT "$line1";
           }
       }
}
2

You are open (OUT, ">out"); in a for loop, which means every single time that the loop iterates, the out file gets recreated anew.

Open the file to write outside the for loop and close it when the loop is finished.

3

Showing us the work that you have done is good. But, just showing us a script that is not working without also showing us sample input and expected output makes it hard to guess at all of the things that might be wrong in your script. The script you have shown us seems to be a strange mix of shell and awk statements.

Please tell us what operating system you're using, tell us what shell you're using, show us some sample input, and show us the exact output you want to produce from that sample input.

4

By the way, perl has an special variable `$.' (that's $ and period) which might help you read the specific line of a file without resorting to making calls to the external head and tail.

So instead of $line1= `head -tmp1 file1| tail -1`;

maybe something like:

open my $fh, '<', $file1 or die "$file1: $!";

my $cntl = 1; # or whatever you want
my $line;
while( <$fh> ) {
    if( $. == $ctl ) { 
        $line = $_;
        last;
    }
}
close $fh

# now you can use $line here
5

@Don Cragun, thank you for remind me about the manner :).
my job is rarely relate to scripting and I just make this script due to my colleague request on yesterday so I just try to use what I know and google what I don't. That's why my script look like a bunch of scraps :o.
by the way, my OS is Linux Redhat, I'm using Perl which included. My script is used for comparing 2 file which contain lines with this info:

 coordinate_x coordinate_y  variable_name  value_1 value_2...

My expect output file which contains all lines with same values, another output file contain lines with different values. Currently, I use loop for comparing 2 file but it's seem take time cause input files having few thousand lines each, so I'm finding another way.

@Aia: thank you so much for your hint. it's just where I stuck :).
Thank you for the suggestion, hope it can reduce the script running time.

6

I still have no idea what you're trying to compare when you're looking at lines from your two input files and I have no idea what output you're trying to produce. Are you just trying to extract lines from your 2nd input file that have a variable_name in the 3rd field that matches a variable_name in the 3rd field on any line in your 1st input file? If this is what you want, it will be easy to write a small, fast awk script to do what you want.

I repeat: Please show us a small, representative pair of sample input files, and show us the output that should be produced from those input files.

7

I want to show you what you're demanding, but I really can't. In my company, pc for working can not access to internet, I have to use another pc which allowed to access internet to go here, and working pc and internet pc are not connected, so there is no way I can show you the infos you want, please pardon me for this.
As I said, my job is rarely relate with scripts. I only know a little C-shell via some scripts for my work verification, know nothing about awk and Perl is just a little since 2 days ago. I use Perl because it can output an excel file (as my colleague told me).

I will try to describe my problem again. I have 2 file which contain lines with these info

coordinate_x coordinate_y variable_name value1 value2 value3

I need to compare which line in 1st file has same variable_name with 2nd file, which doesn't, then print to excel file one sheet for all same variable_name, another sheet for different variable_name.
Currently, I'm stuck in how to compare them quicker, because my script take terrible long running time for compare them. I'm finding another way to overcome my problem.
Please kindly have some suggestion.

Thank you so much in advance :).

8

awk is more concise for these forms of filtering.

awk 'NR==FNR {a[$3]; next} $3 in a' file1 file2 > exist_in_both.file

awk 'NR==FNR {a[$3]; next} !($3 in a)' file1 file2 > does_not_exist_in_both.file

---------- Post updated at 08:47 PM ---------- Previous update was at 08:28 PM ----------

An untested try using perl, which saves two files:
match.txt and unmatch.txt
run it as:
# search.pl file1 file2

#!/usr/bin/perl

use strict;
use warnings;

my $first_file = 1;
my %record;

my $hits = "match.txt";
my $misses = "unmatch.txt";

open my $fh1, '>', "$hits" or die "Could not open $hits: !$\n";
open my $fh2, '>', "$misses" or die "Could not open $misses: !$\n";

while(<>) {
    chomp;
    my @fields = split;
    if ( $first_file == 1 ) {
        $record{$fields[2]} = 1;
        next;
    }
    if (exists $record{$fields[2]}) {
        print $fh1 "$_\n";
    } else {
        print $fh2 "$_\n";
    }
}

continue {
    if (eof) {++$first_file};
}

close $fh1;
close $fh2;
9

Thank you so much for your suggestion, thing's getting much easier now :).

10

If you want three output files, (variables in file1 only, variables in file2 only, and variables in both files) you could also try:

awk '
FNR == NR {
	f1[++n1] = $3
	var[$3]
	next
}
{	if($3 in var) {	print $3 > "in_both"
			delete var[$3]
	} else		print $3 > "in_f2_only"
}
END {	for(i = 1; i <= n1; i++)
		if(f1 in var)
			print f1 > "in_f1_only"
}' file1 file2

This script could be simplified some if you don't care about the order of lines in the output of variable names found only in file1. The way this is currently written variable names in the files in_both and in_f2_only will appear in the order in which they first appeared in file2 and variable names in the file in_f1_only will appear in the order in which they first appeared in file1.

In someone wants to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk , /usr/xpg6/bin/awk , or nawk .

11

Thank you so much, thing is much easier since Aia and you gave suggestions.
Btw, I think I should consider learn more about these scrpit language after this.
Once again, thank you guys so much :).