perl process loop isn't running

I'm trying to figure out why the perl process we have running in a loop isn't working.

Basically its setup to read our queue from Amazon SQS with the results getting inserted into the db.

We are using EC2 for video transcoding and once the conversion takes place our web server hosted outside of Amazon has a perl process that reads SQS.

I'm not a dev but I think I can find the problem if given some suggestions on how to test. I was trying to locate the "aws-daemon" log but wasn't able to find it in /var/log. I see that first line below states to exit if theres an INT flag, but I'm not sure what that means, and if it did set that flag does it mean theres a problem that is still existing somewhere preventing this process from working.

I've rebooted the server, and launched a new ec2 instance but other than that I don't know enough on how to kill or restart a process or whether thats the way to do it. I know there is stuff sitting in the queue waiting for this process to pull it and insert it into the db. Any help would be greatly appreciated.

#!/usr/bin/perl -w
use Error qw(:try);
use MIME::Base64;
use FindBin qw($Bin);
use POSIX qw(setsid setuid);
use Config::Auto;
use Proc::Daemon;
use Log::Log4perl qw(:easy);
use Platform;
use Platform::Queue;
use Platform::Video;
use strict;

# when we get a INT signal, set the exit flag
$SIG{INT} = sub { $::exit = 1 };

# setup logging
Log::Log4perl->easy_init($TRACE);

my ($log, $conf, $video, $sqs, $queue);

$log = Log::Log4perl->get_logger('aws-daemon');
$conf = Config::Auto::parse($Bin . "/aws.conf", format => "colon");
$video = Platform->new();
$sqs = Platform::Queue->new({
  aws_access_key_id     => $conf->{'aws_id'},
  aws_secret_access_key => $conf->{'aws_secret'} });

# fork into the background
# do this first because our process id will change
Proc::Daemon::Init;
$log->trace('calling daemonize()');
&daemonize();

# our infinite loop
while(1) {
  $log->trace('running...');
  exit if $::exit;
  check_queues($conf, $sqs, $video);
  $log->trace('sleeping for two minutes...');
  sleep(120);
  exit if $::exit;
  $log->trace('still running...');
}

sub check_queues {
  $log->trace('check_queues');
  my ($conf, $sqs, $video) = @_;

  try {
    $video->read_result_queue();
  } catch Error with {
    my $ex = shift;
    $log->trace('Exception in check_queues: ' . $ex);
  } finally {
  }
}

sub daemonize {
    chdir '/'                 or die "Can't chdir to /: $!";
    open STDIN, '/dev/null'   or die "Can't read /dev/null: $!";
    open STDOUT, '>>/tmp/awsd' or die "Can't write to /dev/null: $!";
    open STDERR, '>>/tmp/awsd' or die "Can't write to /dev/null: $!";
    defined(my $pid = fork)   or die "Can't fork: $!";
    exit if $pid;
    setsid                    or die "Can't start a new session: $!";
    umask 0;
}

The reason you don't get a log file in /var/log/ is that Log::Log4perl::easy_init will only send messages to stdout/stderr, both of which are redirect to /tmp/awsd in daemonize(). Check that file for any error messages.

The $SIG{INT} construct tells the script what to do should it receive a SIGINT (usually via kill -2 <pid>). It will set the exit variable, which is checked by the loop, which then will exit.

I'm not finding any /tmp/awsd directory currently.

I have a terminal output text file I saved back in June that does show the /tmp/awsd but now the directory shows totally different files

right now the /tmp has
.ICE-unix/ queue-log s3-bash/ .webmin/ .X11-unix/

and in june the /tmp was
awsd php-www.log queue-log s3-bash

---------- Post updated at 01:18 PM ---------- Previous update was at 01:17 PM ----------

So aside from the logs is there something that can be done to execute the process from the cmd line, do I need the pid in order to do that? Any help is really appreciated.

I might recommend a minor rewrite of the sub daemonize function to include some reporting (as well as identifying the correct files that can't be written to...).
Change this around such that:

   chdir '/'                 or die "Can't chdir to /: $!";
   open STDIN, '/dev/null'   or die "Can't read /dev/null: $!";
   open STDOUT, '>>/tmp/awsd' or die "Can't write to /tmp/awsd: $!";
   open STDERR, '>>/tmp/awsd' or die "Can't write to /tmp/awsd: $!";
   defined(my $pid = fork)   or die "Can't fork: $!";
#   exit if $pid;
   if ($pid)
   {
      print "stuff that is relevant to the process is output to screen"
      exit
   }
   setsid                    or die "Can't start a new session: $!";
   umask 0;



This will let you know what it sees BEFORE you exit - it might help you to debug.

You might want to read up on the perl module "Log::Log4perl" - it might have some information that would make this more clear.

The /tmp/aswd file was probably deleted when you rebooted, since most Linux distributions will clean up /tmp on boot.

To start the daemon: /path/to/aws-daemon
To stop the daemon:

$ ps -ef | grep aws-daemon # Note the PID
$ kill -2 <pid of aws-daemon>

Generally, when a daemon has to be restarted, it isn't necessary to reboot the whole server (as opposed to some services in Windows)

It gave an error:

Can't locate Platform.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.8 /usr/local/share/perl/5.8.8 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at /path/to/aws-daemon line 9.
BEGIN failed--compilation aborted at /path/to/aws-daemon line 9.

It appears that you are missing a perl module or two...

line 9#    use Platform;
line 10#   use Platform::Queue;
line 11#   use Platform::Video;

I assume that you have a Platform.pm available somewhere on your system. There are a few ways to include it, but the easiest in my view is to add the following at the top of your script...

#!/usr/bin/perl -w

BEGIN {
push @INC, "<directory containing Platform.pm>";
}
.
.

This dynamically adds the directory to the @INC variable before any of the main code runs and makes it clear what you're doing. You can do the same thing by adding a -I option on the command line but then you have to invoke Perl explicitly rather than just executing your .pl script

they are there, though I'm not sure what "missing in @INC" means because they are in the same directory as the aws-daemon

would it say its missing if the process wasn't running?

the only thing that happened at the same time this occured was the data center where this is hosted experienced an outage

Having them listed in your perl script is not the same as having the modules installed on your system somewhere.

The @INC is a list of directories where perl will look to find these modules.

(@INC contains:
/etc/perl
/usr/local/lib/perl/5.8.8
/usr/local/share/perl/5.8.8
/usr/lib/perl5
/usr/share/perl5
/usr/lib/perl/5.8
/usr/share/perl/5.8
/usr/local/lib/site_perl
.)

It wouldn't hurt to run a simple command to see if the file exists and is simply not in the path:

find / -name Platform.pm -print

Yes, Platform.pm is on the system

I don't really want to add the @INC variable at this time because I'm completely sure what even caused the perl process to stop, if it was due to the network outage then thats fine I just want to know that I'm not trying to initiate something that had been terminated because of another problem in the system.

I thought I had a disaster recovery plan so that I could reboot the system entirely myself in a matter of a few short steps but I think we blew that off or forgot about it. My sysadmin is off onto another adventure and unreachable so I'm trying my best to not do more harm than good.

---------- Post updated at 04:39 PM ---------- Previous update was at 04:25 PM ----------

i ran that and its in the same directory

/same/path/to/aws-daemon

How does @INC provide the path, in other words it looks like it provides the path to

/etc/perl
/usr/local/lib/perl/5.8.8
/usr/local/share/perl/5.8.8
/usr/lib/perl5
/usr/share/perl5
/usr/lib/perl/5.8
/usr/share/perl/5.8
/usr/local/lib/site_perl

So shouldn't the /same/path/to/Platform.pm be in that list

---------- Post updated at 04:56 PM ---------- Previous update was at 04:39 PM ----------

Ok, so I ran:

perl -e "print join(\"\n\", @INC);"

and the /path/to/Platform.pm is not in the results this is all thats there:

/etc/perl
/usr/local/lib/perl/5.8.8
/usr/local/share/perl/5.8.8
/usr/lib/perl5
/usr/share/perl5
/usr/lib/perl/5.8
/usr/share/perl/5.8
/usr/local/lib/site_perl

Is it possible that when the reboot occured it upgraded to 5.8.8 and wiped the @INC array clean?

If thats the case, then this /path/to/Platform.pm may just be one of many that are now missing in the @INC array, so if I just manually add it back in via command line as:

perl -I /path/to/Platform.pm

Now I also see a Platform.pm.old in the same directory but not sure if that is of any help.

---------- Post updated at 08:03 PM ---------- Previous update was at 04:56 PM ----------

I tried doing the perl -I command and it didn't seem to add it to the @INC list

---------- Post updated at 09:57 PM ---------- Previous update was at 08:03 PM ----------

Ok another discovery :rolleyes:

when I login to root with su the /Platform.pm and the /aws.conf are not there

is that what the problem is? when I'm logged in as a regular user they are there.

---------- Post updated 09-02-09 at 03:39 PM ---------- Previous update was 09-01-09 at 09:57 PM ----------

Added the following to the top of the script, which worked.

use lib '/path/to/modules';