save every line in log file with matched string

meggae · March 22, 2009, 8:24am

i have been doing this script to match every line in a current log file (access_log) with strings that i list from a path (consist of 100 of user's name ex: meggae )..
and then make a directory of every string from the text file (/path/meggae/) --->if it matched..
then print every line from the log file if it matched the string into a new file name base on the string (meggae.txt) in the new created directory (/path/meggae/meggae.txt)..

below is the example script that i made..it has no error so far but it uses 100% of my processes as well as my memory and sometimes it uses my swap space and my lappy go lagged and hang..i hope some explanation and some solution can help me not to crash my dear lappy

#!/usr/bin/perl -w
use File::Basename;
use CGI;
use CGI::Carp qw(fatalsToBrowser);
use File::Path;
use File::Copy;
print "Content-type: text/html\r\n";
print "\r\n";

open (example7, "/path/foo/foo/access_log") || die ("Could not open file. <br> $!");
@cisco = <example7>;

$new = qx(ls /path/foo/) or die "Could not read file. <br> $!"; #contain user's name ex: meggae and other 100 users

@passwd = split(' ',$new); #to split every string from <space> char
print $passwd; # try to see if it works and it worked well

foreach $string (sort(@passwd)) {

my $path = "/path/foo/foo/$string/log";

if (! -d $path)
{
mkpath($path,755) or die "Failed to create $path: $!\n";
}

foreach $line (sort(@cisco))
{

if ($line =~ $string)
{
foreach ($line =~ $string)
{
$file = "$string.txt";
$append = 0;
if ($append){
open (OUTFILE,">$file");
}else{
open (OUTFILE,">>$file");
}
foreach $line (@cisco)
{
foreach ($line =~ $string)
{print OUTFILE $line;}
}
close (OUTFILE);
}
foreach ($path =~ $string) {
move($file,$path) or die "Failed to copy $file: $!\n"; }

}

}
}

any solutions will tremendously help me with this space/memory crashing and what have i done wrong

rikxik · March 22, 2009, 10:52pm

Too many un-necessary nested foreach. From what I can see, the outer foreach for each entry in password and the inner foreach for each line in cisco is sufficient. Why do you have to repeatedly keep doing foreach even for opening and closing files?

What do you mean by "foreach ($line =~ $string)"? Why can't you write it as "if($line =~ $string)" and do what you need to do within the if block?

There are many other issues but you may want to sort out the above problems first and see if that makes the process less demanding.

insamniac · March 24, 2009, 12:05pm

Here's a way you could do it in ksh.. not the most efficient way, but worth a shot to see if it eats up your system resources as badly as your method..

#!/bin/ksh

logfile=/path/to/logfile
namelist=/path/to/namelist
outputdir=/output/dir


cat $namelist | while read name
do

   grep $name $logfile > temp
   if [ $(cat temp | wc -l) -ne 0 ];then 
    
     mkdir  ${outputdir}/$name
     cat temp > ${outputdir}/$name
     cat temp > ${outputdir}/${name}/${name}.txt
       
    fi

done

meggae · April 6, 2009, 11:38pm

rikxik..
thanks for the brilliant suggestion given..i also thought the problem might be with the programming (foreach statement) as i'm not into much of the programming..
i've review the code and cut out the unnecessary loop..it works and it works great if i execute it from the terminal instead of browser..
thanks insamniac for ur ksh opinion..i would really like to consider it but i don't really have much time now to learn new things..but i currently trying the code already..it surely gives me another sight way of solving coming problems..

#!/usr/bin/perl -w

use File::Basename;
use CGI;
use CGI::Carp qw(fatalsToBrowser);
use File::Path;
use File::Copy;
print "Content-type: text/html\r\n";
print "\r\n";

$accessfile = "/var/log/foo/access_log";
open (example7, "/var/log/foo/access_log") || die ("Could not open file. <br> $!");

@cisco = <example7>;

@passwd = split(' ',$new);

open(READER, $accessfile) or die "canot read log:$accessfile:$!";
@LOGFILE = <READER>;

while(<READER>) {
($machine, $date, $time, $tzone, $method, $url, $httpvers, $response, $size) = /([^\s]) [^\s] [^\s]* \[([^\:]*):([^\s]) ([^\]])\] \"([^\s]) ([^\s]) HTTP\/([^\"])\" (\d) (\d*)/;

($day, $month, $year) = split('/',$date);
\}
foreach $string (sort(@passwd)) {

my $path = "/path/foo/foo/$string/log";

if (! -d $path)
{
mkpath($path,755) or die "Failed to create $path: $!\n";
}

$file = "$string.txt";

if ($path =~ $string) {
foreach $line (sort(@cisco)) {
$append = 0;
if ($append){
open (OUTFILE,">$file"); if ($line =~ $string) {print OUTFILE $line;}
}else{
open (OUTFILE,">>$file"); if ($line =~ $string) {print OUTFILE $line;}
} }
close (OUTFILE);
move($file,$path) or die "Failed to copy $file: $!\n";
}

}
close (example7);