Perl script to rotate logs

theninja · March 12, 2008, 12:17pm

I have a shell script that will gzip/tar/archive application logs that are over 20 days old which works just fine, but I would like to convert to a Perl script. Problem is, I'm a beginner with Perl and all attempts so far have failed.
Basicaly I have a log dir /app/logs that contains several different logs files that are created daily. One log for each day.
this.log.2008.02.01
this.log.2008.02.02
this.log.2008.02.03
that.log.2008.02.01
that.log.2008.02.02
that.log.2008.02.03
and so on......

the script finds any log over 20 days old, gzips it, the moves it to /app/logs/archive, after 60 days these files are tar'd up and moved to /log_archive/app_logs

Any help would be greatly appreciated.

here is the shell script.....

#!/usr/bin/ksh

# Gzip is located in /usr/contrib/bin
export PATH=$PATH:/usr/contrib/bin

# Last 2 digits of the current year
curr_year=`date +%o`

# Directorys
LOG_DIR=/app/logs
AUDIT_DIR=/app/logs/audit
ARCHIVE_DIR=/app/logs/archive
LOG_ARCHIVE_DIR=/log_archive/app_logs

# Number of days before things are gzip'd.
COMPRESS_DAYS=7
# Number of days till gzips are added to a tar file.
MV_AR_ARCHIVE_DAYS=7
# Number of days until the tar files are moved to log partition.
MV_MOAR_ARCHIVE_DAYS=60
# Number of days till tars are removed from log partition.
RM_MOAR_ARCHIVE_DAYS=180

#############

MOAR_HOST_NAME=$(hostname|tr [:lower:] [:upper:])

COMPAREYEAR="`date '+%y'`"
DATEYEAR="`date '+%Y'`"
DATEMONTH="`date '+%m'`"
DATEMONTH=$(( $DATEMONTH - 1 ))

if [ $DATEMONTH -eq 0 ]
then
        DATEMONTH=12
        DATEYEAR=$(( $DATEYEAR - 1 ))
fi
if [ $DATEMONTH -lt 10 ]
then
        DATEMONTH=0$DATEMONTH
fi

# DATESTAMP is the last month
DATESTAMP=$DATEYEAR.$DATEMONTH
# echo $DATESTAMP

#change to the log directory
cd ${LOG_DIR}
echo "Compressing Logs files"
find *fin*log* -prune -mtime +${COMPRESS_DAYS} -exec gzip {} \;
find this* -prune -mtime +${COMPRESS_DAYS} -exec gzip {} \;
find that* -prune -mtime +${COMPRESS_DAYS} -exec gzip {} \;
#move GZIP files to archive directory
find *.gz -mtime +${MV_AR_ARCHIVE_DAYS} -exec mv {} ${LOG_ARCHIVE_DIR}/ \;

#move GZIP files to archive directory
find *.gz -mtime +${MV_AR_ARCHIVE_DAYS} -exec mv {} ${ARCHIVE_DIR}/  \;

#change to the Archive Directory
cd ${ARCHIVE_DIR}
echo "Adding GZIP files to tar archive"
if [ ! -f "arlogs_${DATESTAMP}.tar" ]
then
        touch place.txt
        tar -cf arlogs_${DATESTAMP}.tar place.txt
fi

#loop for all the files in the directory.
ls -l *gz |\
while read file; do
    # Get month
    mo=`echo $file |awk '{print $6}'`
    case $mo in
        Jan) month=01;;
        Feb) month=02;;
        Mar) month=03;;
        Apr) month=04;;
        May) month=05;;
        Jun) month=06;;
        Jul) month=07;;
        Aug) month=08;;
        Sep) month=09;;
        Oct) month=10;;
        Nov) month=11;;
        Dec) month=12;;
          *) echo "Error:$prog:cannot determine month"; exit 1;;
    esac

    # Get day
    day=`echo $file |awk '{print $7}'`
    if [ -z "$day" ]; then
        echo "Error:$prog:cannot determine day"
        exit 1
    fi
    if [ $day -lt 10 ]; then
        day="0$day"
    fi

    # Get year
    file_year=`echo $file |awk '{print $8}'`
    if [ `echo $file_year |grep -c ":"` -gt 0 ]; then
        year=$curr_year
    else
        year=`echo $file_year |sed 's/^..//'`
    fi
    if [ -z "$year" ]; then
        echo "Error:$prog:cannot determine year"
        exit 1
    fi

    # File modification time (timestamp) in xx/xx/xx format
    timestamp="$month/$day/$year"

    # Get filename from long listing
    file=`echo $file |awk '{print $9}'`

    # Add to tar file and remove gzip file
    case $timestamp in
        *$DATESTAMP*)
            tar -uf arlogs_${DATESTAMP}.tar ${file}
            if [ $? -eq 0 ]; then
                rm ${file}
            fi ;;
        $DATEMONTH/*/$COMPAREYEAR)
            tar -uf arlogs_${DATESTAMP}.tar ${file}
            if [ $? -eq 0 ]; then
                rm ${file}
            fi ;;
    esac
done

#check to see if directory on archive partition is there
if [ ! -d "${LOG_ARCHIVE_DIR}/${MOAR_HOST_NAME}_logs_${DATEYEAR}" ]
then
  mkdir ${LOG_ARCHIVE_DIR}/${MOAR_HOST_NAME}_logs_${DATEYEAR}
fi

# Move from the archive directory to the archive partition
find *.tar -mtime +${MV_MOAR_ARCHIVE_DAYS} -exec mv {} ${ARCHIVE_DIR}/${MOAR_HOST_NAME}_logs_${DATEYEAR} \;

#change to the archive partition directory
cd ${LOG_ARCHIVE_DIR}/${MOAR_HOST_NAME}_logs_${DATEYEAR}

# remove any old tars
find *.tar -mtime +${RM_MOAR_ARCHIVE_DAYS} -exec rm {} \;

Thanks
Theninja

theninja · March 17, 2008, 10:26am

anybody, anybody, Bueller, Beuller ?

ShawnMilo · March 17, 2008, 11:42am

Posting a working script and basically saying "re-write this for me in Perl" won't work. Maybe that's not how you meant it, but you really didn't give us much to work with.

There are probably no replies because nobody wants to do all your work for you. You say all your attempts have failed. Fair enough, but post your best attempt, or pieces which are failing. I'm sure plenty of people will be willing to help you fix your code.

ShawnMilo

theninja · March 18, 2008, 10:57am

My apologies, I definitley did not mean for someone to re-write it for me, I wouldn't learn anything that way. Here is the test code that is not working for me, it doesn't do anything. No output, no log changes, just sits there until I Ctrl-c out of it.

#!/usr/local/bin/perl

$LOGPATH='/home/logs';          # Location of AR Log files
$ARCHIVE_DIR='/home/log_archive/moar_logs';     # Location of archived logs
$MONTH=`date +%m`;                      # Current Month
$DAY=`date +%d`;                        # Current Day of month
$MAXCYCLE='2';                          # Number of days to keep log files
$GZIP='/usr/contrib/bin/gzip';          # Location of gzip for compression

@LOGNAMES=('twiz*.log*','nestl*','riesen*');
%ARCHIVE=('*.$MONTH.*'=>1);

chdir $LOGPATH;  # Change to the log directory
foreach $filename (@LOGNAMES) {
  my $oldest = "$filename.$MAXCYCLE";
  archive($oldest) if -e $oldest and $ARCHIVE{$filename};
  for (my $s=$MAXCYCLE; $s >= 0; $s-- ) {
        $oldname = $s ? "$filename.$s" : $filename;
        $newname = join(".",$filename,$s+1);
        rename $oldname,$newname if -e $oldname;
    }
}
kill 'HUP',`cat $PIDFILE`;

printf "$LOGPATH\n";
printf "$ARCHIVE_DIR\n";
printf $MONTH;
printf $DAY;
printf "$MAXDAYS\n";
printf "$GZIP\n"

system · March 18, 2008, 11:17am

@LOGNAMES=('twiz*.log*','nestl*','riesen*');
%ARCHIVE=('*.$MONTH.*'=>1);

Perl doesn't expand the wildcards and metacharacters for you here like the shell would.

It would seem like a good idea to add print statements inside the loop to see where it gets stuck.

chdir $LOGPATH;  # Change to the log directory
foreach $filename (@LOGNAMES) {
  my $oldest = "$filename.$MAXCYCLE";
print "DEBUG: oldest is '$oldest'\n";
  archive($oldest) if -e $oldest and $ARCHIVE{$filename};
print "DEBUG: '$oldest' exists\n" if -e $oldest;
print "DEBUG: ARCHIVE{'$filename'} is true\n" if $ARCHIVE{$filename};
print "DEBUG: would archive oldest\n" if -e $oldest and $ARCHIVE{$filename};
  for (my $s=$MAXCYCLE; $s >= 0; $s-- ) {
print "DEBUG: s is $s\n";
        $oldname = $s ? "$filename.$s" : $filename;
        $newname = join(".",$filename,$s+1);
print "DEBUG: oldname now '$oldname', newname is '$newname'\n";
print "DEBUG: would rename '$oldname' to '$newname'\n" if -e $oldname;
        rename $oldname,$newname if -e $oldname;
    }
}

I often omit the indentation on debug statements to make it easy to remove them later. A more sustainable solution is to make the prints conditional on a $debug variable, and leave them in.

ShawnMilo · March 18, 2008, 1:14pm

It seems like part of the problem is that the script isn't looking at the files at all. It's looping through strings which you intend to represent filename patterns, but it's never looking at the directory.

Hopefully this will get you started:


#!/usr/local/bin/perl

opendir(DIR, ".");

@files = grep { /^(twiz*.log*|nestl*|riesen*)/ } readdir(DIR);
closedir(DIR);

foreach $file  (@files){

    print "$file\n";

}

Also, I don't see where the "archive()" function you're calling has been declared. Maybe it's built into your environment. If not, maybe that's what's hanging up.

ShawnMilo

kahuna · March 18, 2008, 2:38pm

Try

@LOGNAMES=map { glob($_) } 'twiz*.log*','nestl*','riesen*';

theninja · March 20, 2008, 12:41pm

as suggested I added/modified the code as follows

@LOGNAMES=map { glob($_) } 'Proxy*','Claim*','Eligibility*';
# @LOGNAMES=('fin*.log*','Proxy*','Claim*','Eligibility*');
%ARCHIVE=('*.$MONTH.*'=>1);

chdir $LOGPATH;  # Change to the log directory
printf "$LOGPATH\n";
foreach $filename (@LOGNAMES) {
  my $oldest = "$filename.$MAXCYCLE";
printf $filename\n;
print "DEBUG: oldest is '$oldest'\n";
  archive($oldest) if -e $oldest and $ARCHIVE{$filename};
print "DEBUG: '$oldest' exists\n" if -e $oldest;
print "DEBUG: ARCHIVE{'$filename'} is true\n" if $ARCHIVE{$filename};
print "DEBUG: would archive oldest\n" if -e $oldest and $ARCHIVE{$filename};
  for (my $s=$MAXCYCLE; $s >= 0; $s-- ) {
print "DEBUG: s is $s\n";
        $oldname = $s ? "$filename.$s" : $filename;
        $newname = join(".",$filename,$s+1);
print "DEBUG: oldname now '$oldname', newname is '$newname'\n";
print "DEBUG: would rename '$oldname' to '$newname'\n" if -e $oldname;
        rename $oldname,$newname if -e $oldname;

Whe I run it, it only prints out the $LOGPATH then hangs with no output. So i am asuming that one of these two lines is failing....
foreach $filename (@LOGNAMES) {
my $oldest = "$filename.$MAXCYCLE";

Perls doesn't seem very easy to debug, no stdout or stderr?

kahuna · March 20, 2008, 1:41pm

The code posted doesn't compile on my machine. Try changing

printf $filename\n;

printf "$filename\n";

And adding 2 closing curly brackets - "}"

KevinADC · March 20, 2008, 1:48pm

Not sure what this is supposed to do:

%ARCHIVE=('*.$MONTH.*'=>1);

archive() is not a perl function, not sure what that does unless you have loaded a module that has that function imported into your perl program.

This line is wrong:

printf $filename\n;

\n will not be interpolated properly when unquoted:

printf "$filename\n";

printf in perl is for writing to files or adding formatting, you should just use print instead of printf when printing to stdout. Add the warnings pragma to your script:

use warnings;

and it will alert you to problems and potential problems.

theninja · March 20, 2008, 2:33pm

My bad, I guess I should post the entire thang.... This is what I am testing

#!/usr/local/bin/perl

$LOGPATH='/home/logs';          # Location of Log files
$ARCHIVE_DIR='/home/log_archive/moar_logs';     # Location of AR archived logs
$MONTH=`date +%m`;                      # Current Month
$DAY=`date +%d`;                        # Current Day of month
$MAXCYCLE='2';                          # Number of days to keep log files
$GZIP='/usr/contrib/bin/gzip';          # Location of gzip for compression

@LOGNAMES=map { glob($_) } 'fin**'twiz*.log*','nestl*','riesen*';
# @LOGNAMES=('fin*.log*','twiz*.log*','nestl*','riesen*');
%ARCHIVE=('*.$MONTH.*'=>1);

chdir $LOGPATH;  # Change to the log directory
print "$LOGPATH\n";
foreach $filename (@LOGNAMES) {
  my $oldest = "$filename.$MAXCYCLE";
print "$filename\n";
print "DEBUG: oldest is '$oldest'\n";
  archive($oldest) if -e $oldest and $ARCHIVE{$filename};
print "DEBUG: '$oldest' exists\n" if -e $oldest;
print "DEBUG: ARCHIVE{'$filename'} is true\n" if $ARCHIVE{$filename};
print "DEBUG: would archive oldest\n" if -e $oldest and $ARCHIVE{$filename};
  for (my $s=$MAXCYCLE; $s >= 0; $s-- ) {
print "DEBUG: s is $s\n";
        $oldname = $s ? "$filename.$s" : $filename;
        $newname = join(".",$filename,$s+1);
print "DEBUG: oldname now '$oldname', newname is '$newname'\n";
print "DEBUG: would rename '$oldname' to '$newname'\n" if -e $oldname;
        rename $oldname,$newname if -e $oldname;
    }
}
kill 'HUP',`cat $PIDFILE`;

 sub archive {
        my $f = shift;
        my $base = $f;
        $base =~ s/\.\d+$//;
        my $fn = strftime("$base.%Y-%m-%d_%H:%M.gz.idea",localtime);
        system "$GZIP -9 -c $f | $GZIP -kfile $MAP > $fn";
        system "$TAR rvf $base.tar --remove-files $fn";
 }

This prints out the $LOGPATH and hangs......
I am pretty sure I have this wrong, but hey it's a learning experience. Making for long frustrating days like I like.

kahuna · March 20, 2008, 3:48pm

This still doesn't compile. Your quotes don't match up.

@LOGNAMES=map { glob($_) } 'fin**'twiz*.log*','nestl*','riesen*';

Are you sure you have files in the current directory that match the wild carding? Try adding file "abctest" to the current directory, then replace the above with

@LOGNAMES=map { glob($_) } 'fin*','*twiz*.log*','nestl*','riesen*', '*abc*';

As KevinADC mentioned, it would probably help to have

#!/usr/local/bin/perl
use warnings;

at the top of your script

theninja · March 20, 2008, 4:04pm

use warnings is worrking well, now to debugg...

./logrotate.pl
Name "main::TAR" used only once: possible typo at ./logrotate.pl line 43.
Name "main::PIDFILE" used only once: possible typo at ./logrotate.pl line 35.
Name "main::MAXDAYS" used only once: possible typo at ./logrotate.pl line 50.
/home/p029052/logs
Use of uninitialized value in print at ./logrotate.pl line 17.
HASH(0x40010f78)
DEBUG: oldest is 'HASH(0x40010f78).2'
DEBUG: s is 2
DEBUG: oldname now 'HASH(0x40010f78).2', newname is 'HASH(0x40010f78).3'
DEBUG: s is 1
DEBUG: oldname now 'HASH(0x40010f78).1', newname is 'HASH(0x40010f78).2'
DEBUG: s is 0
DEBUG: oldname now 'HASH(0x40010f78)', newname is 'HASH(0x40010f78).1'
Use of uninitialized value in concatenation (.) or string at ./logrotate.pl line 35.

Thanks for the help everyone, I am getting somewhere now. I will post what I get after debugging.

theninja · March 20, 2008, 5:24pm

Ok I am almost there, only two issues left I think...

$PIDFILE format not correct for Perl? Currently commented out because it bombed.
Not picking up the logfile names....see output (print $map does not pick up the lognames) I did move the script into the log directory where the logs reside.

#!/usr/local/bin/perl
use warnings;

# $PIDFILE=`ps -aefx | grep logrot | grep -v grep |awk '{print $2}'`;
$LOGPATH='/home/logs';          # Location of Log files
$ARCHIVE_DIR='/home/log_archive/logs';     # Location of archived logs
$MONTH=`date +%m`;                      # Current Month
$DAY=`date +%d`;                        # Current Day of month
$MAXCYCLE='2';                          # Number of days to keep log files
$GZIP='/usr/contrib/bin/gzip';          # Location of gzip for compression
$TAR='/usr/bin/tar';                    # Location of tar command

@LOGNAMES=map {glob($_) } {'fin*.log*','*twiz*.log*','nestl*','riesen*'};
%ARCHIVE=('*.$MONTH.*'=>1);
# print "$PIDFILE\n";
chdir $LOGPATH;  # Change to the log directory
printf "$LOGPATH\n";
print "$map";
foreach $filename (@LOGNAMES) {
  my $oldest = "$filename.$MAXCYCLE";
printf "$filename\n";
print "DEBUG: oldest is '$oldest'\n";
  archive($oldest) if -e $oldest and $ARCHIVE{$filename};
print "DEBUG: '$oldest' exists\n" if -e $oldest;
print "DEBUG: ARCHIVE{'$filename'} is true\n" if $ARCHIVE{$filename};
print "DEBUG: would archive oldest\n" if -e $oldest and $ARCHIVE{$filename};
  for (my $s=$MAXCYCLE; $s >= 0; $s-- ) {
print "DEBUG: s is $s\n";
        $oldname = $s ? "$filename.$s" : $filename;
        $newname = join(".",$filename,$s+1);
print "DEBUG: oldname now '$oldname', newname is '$newname'\n";
print "DEBUG: would rename '$oldname' to '$newname'\n" if -e $oldname;
        rename $oldname,$newname if -e $oldname;
    }
}
#kill 'HUP',`cat $PIDFILE`;

 sub archive {
        my $f = shift;
        my $base = $f;
        $base =~ s/\.\d+$//;
        my $fn = strftime("$base.%Y-%m-%d_%H:%M.gz.idea",localtime);
        system "$GZIP -9 -c $f | $GZIP $map > $fn";
        system "$TAR rvf $base.tar --remove-files $fn";
 }

printf "$LOGPATH\n";
printf "$ARCHIVE_DIR\n";
printf $MONTH;
printf $DAY;
printf "$MAXCYCLE\n";
printf "$GZIP\n";
printf "$TAR\n"

here is the output of above code

/home/logs
Use of uninitialized value in concatenation (.) or string at ./logrotate.pl line 19. (line 19 = print "$map"

HASH(0x40010f78)
DEBUG: oldest is 'HASH(0x40010f78).2'
DEBUG: s is 2
DEBUG: oldname now 'HASH(0x40010f78).2', newname is 'HASH(0x40010f78).3'
DEBUG: s is 1
DEBUG: oldname now 'HASH(0x40010f78).1', newname is 'HASH(0x40010f78).2'
DEBUG: s is 0
DEBUG: oldname now 'HASH(0x40010f78)', newname is 'HASH(0x40010f78).1'
/home//logs
/home/log_archive/ogs
03
20
2
/usr/contrib/bin/gzip
/usr/bin/tar

Thanks everyone almost there.

kahuna · March 21, 2008, 7:30am

What exactly do you mean by bombed? Didn't compile? Didn't produce the correct result? Resulted in an error? Did you try running the command from the shell prompt?

Perl is filled with traps for the unwary.

@LOGNAMES=map {glob($_) } {'fin*.log*','*twiz*.log*','nestl*','riesen*'};

Please stick with the original syntax. It it not helpful to post one thing, then run something different and complain that it doesn't work.

@LOGNAMES=map {glob($_) } 'fin*.log*','*twiz*.log*','nestl*','riesen*';

theninja · March 21, 2008, 11:11am

Sorry, It would not compile. I have fixed that issue though by escaping out the $2.

$PIDFILE=`ps -aefx | grep logrot | grep -v grep |awk '{print \$2}'`;

Still having trouble with the LOGNAMES. But i am getting there.

No way i can stick with the original syntax, it has changed a lot. My last post has the current syntax with the above change in it.

kahuna · March 21, 2008, 11:32am

By original syntax, I meant for the specific line that was posted (@LOGNAMES). You added "{" and "}" which changes the way perl interprets the line. Please use the line below.

@LOGNAMES=map {glob($_) } 'fin*.log*','*twiz*.log*','nestl*','riesen*';

theninja · March 21, 2008, 11:51am

You are correct kahuna, took out those two curly braces and it is working.
Now for a little fine tuning and I'll put it through a full test.

Great learning experience

THANK YOU ALL.
theninja

KevinADC · March 21, 2008, 3:06pm

I'd still like to know what this line is supposed to be for?

%ARCHIVE=('*.$MONTH.*'=>1);

wrapping a perl scalar in single-quotes is going to kill the variable expansion and treat it as a litteral string. And the "."'s are not interpolated as concatenation operators if thats what you were expecting, although it is unlcear what you are expecting.