Determine differient IP in a very long record.

Shell Programming and Scripting
1

Hi Everyone,

I have a txt file with 40k lines.
111 1.1.1.1
111 1.1.1.1
111 2.2.2.2
111 1.1.1.1
111 1.1.1.1

How would use perl to easy have IP list, the final result will be:
1.1.1.1,2.2.2.2,

I can only think using if to determine one by one through all 40k, this is working, but thought not efficient. Please advice.

Thanks

2 
perl -lane'
  push @f, $F[1] unless $x{$F[1]}++;
  print join ",", @f if eof
  ' infile

Or if you really want the final ,:

perl -ane'
  print $F[1], "," unless $x{$F[1]}++;
  print $/ if eof
  ' infile
3

Thanks radoulov, it works ;).

I have another file:
312009 1 0620233743 0620233748 20090620235854 20.7.22.14 aaa
312009 2 0620233743 0620233748 20090620235854 20.7.22.14 aaa
312009 3 0620233743 0620233748 20090620235854 20.7.22.14 aaa
312009 5 0620233743 0620233748 20090620235854 21.7.22.14 aaa
312009 2 0620233743 0620233748 20090620235854 20.7.22.14 aaa

I would like to hav a result of:
20.7.22.1 8
21.7.22.1 5

i did:
perl -F' ' -lane '$h{$F[5]} += $F[1]; END{print "$: $h{$}" for sort keys %h}' a.txt

But failed, please advice :confused: Thanks

Instead of
[root@localhost ~]# perl -ane'
print $F[1], "," unless $x{$F[1]}++;
print $/ if eof
' a.txt

if i have a file a.pl, how can i put this command into that file?

Thanks

4

You mean 20.7.22.14, not 20.7.22.1, rigth?

perl -ane'
  $x{$F[5]} += $F[1];
  print map "$_ $x{$_}\n", sort { $x{$b} <=> $x{$a} } keys %x
    if eof
  ' infile
5

Sorry sorry, yes it is 20.7.22.14

6

Use this code as a the Perl script:

#!/usr/bin/env perl

use warnings;
use strict;

my %x;

while (<>) {
  my @F = split;
  $x{$F[5]} += $F[1];
}

print map "$_ $x{$_}\n", 
  sort { $x{$b} <=> $x{$a} } keys %x;
7

Thanks radoulov :b: it works so perfect just few seconds to process my file, inside has 30k lines.

If i have a.pl, how would I transfer this code into that perl file? :frowning: Thanks

8

See the addition in my previous post. Use it like this:

./scriptname inputfile
9

Thanks :smiley:

---------- Post updated at 02:29 PM ---------- Previous update was at 02:11 PM ----------

Hi Radoulov,

[root@localhost ~]# cat aaaa
312009 1 0620233743 0620233748 20090620235854 20.7.22.14 aaa
312009 2 0620233743 0620233748 20090620235854 20.7.22.14 aaa
312009 3 0620233743 0620233748 20090620235854 20.7.22.14 aaa
312009 5 0620233743 0620233748 20090620235854 21.7.22.14 aaa
312009 2 0620233743 0620233748 20090620235854 20.7.22.14 aaa
[root@localhost ~]# cat a.pl
#!/usr/bin/perl
use strict;
use warnings;
my %x;
my @F;
open(my $FB, "/root/aaaa") or die "$!";
while (<$FB>) {
  my @F = split;
  $x{$F[5]} += $F[1];
}
print map "$_ $x{$_}\n", sort { $x{$b} <=> $x{$a} } keys %x;
close $FB;
[root@localhost ~]# ./a.pl
20.7.22.14 8
21.7.22.14 5
[root@localhost ~]#

If I would like to have the result of:
20.7.22.14 8 4
21.7.22.14 5 1
Total: 13

##############
4 and 1 is kind of "cat aaaa | grep 20.7.22.14 | wc -l".
Total: 13 is 8+5=13.

Please advice how i can add this new result into your script. :confused: Thanks

10

Found the solution aly

[root@localhost ~]# cat a.pl
#!/usr/bin/perl
use strict;
use warnings;
my %x;
my %xx;
my @F;
open(my $FB, "/root/aaaa") or die "$!";
while (<$FB>) {
my @F = split;
$x{$F[5]} += $F[1];
$xx{$F[5]} += 1;
}
print map "$_ $xx{$_} $x{$_}\n", sort { $x{$b} <=> $x{$a} } keys %x;
close $FB;

So $xx value is the one to count how many lines.

11

Or just:

++$xx{$F[5]};

Glad you found the solution yourself.