Problems with log rotate and virtual hosts

anderow · August 4, 2008, 4:58pm

ok guys,

Im running Centos 4.6 Final
latest versions of apache, mysql etc etc

Basically, I have several websites on there and one of them is a busy forum and it generates big log files in not that long of a timeperiod.

I set it up about 8 months ago so that logrotate was rotating the logs of my virtual hosts and pruning them at a set size limit, then renaming them as it went along so they were log 1, log 2, log 3 and so on.

This was working great after quite a bit of testing and tweaking but I found out tonight that for some reason this has stopped working and one of the logs is now over 5GB!

I would delete it and start over but I need the information in it for awstats so thats not really an option. Which brings me to my next point!!

I installed Awstats maybe 3 months ago and got it working, but Im not sure if that is what has broken my logrotate script?

Now I know this DID work perfectly!!

but I would appreciate some help in the best way to trouble shoot the problem.

I have restarted Cron.d

and also forced logrotate to run, and go thits

error: httpdvirtual:4 unexpected text
Create/Update database for config "/etc/awstats/awstats.site1.conf" by AW Stats version 6.6 (build 1.887)
From data in log file "/var/log/httpd/site1/site1accesslog" ...
Phase 1 : First bypass old records, searching new record...
Direct access after last parsed record (after line 7947985)
Jumped lines in file: 7947985
Found 7947985 already parsed records.
Parsed lines in file: 5351
Found 243 dropped records,
Found 0 corrupted records,
Found 0 old records,
Found 5108 new qualified records.
Create/Update database for config "/etc/awstats/awstats.site2.conf" by AWStats version 6.6 (build 1.887)
From data in log file "/var/log/httpd/site2/site2.accesslog"...
Phase 1 : First bypass old records, searching new record...
Direct access after last parsed record (after line 7556)
Jumped lines in file: 7556
Found 7556 already parsed records.
Parsed lines in file: 0
Found 0 dropped records,
Found 0 corrupted records,
Found 0 old records,
Found 0 new qualified records.
Create/Update database for config "/etc/awstats/awstats.site3.conf" by AWSta ts version 6.6 (build 1.887)
From data in log file "/var/log/httpd/site3/site3.accesslog"...
Phase 1 : First bypass old records, searching new record...
Direct access after last parsed record (after line 444693)
Jumped lines in file: 444693
Found 444693 already parsed records.
Parsed lines in file: 94
Found 0 dropped records,
Found 0 corrupted records,
Found 0 old records,
Found 94 new qualified records.
Create/Update database for config "/etc/awstats/awstats.localhost.localdomain.co nf" by AWStats version 6.6 (build 1.887)
From data in log file "/var/log/httpd/access_log"...
Phase 1 : First bypass old records, searching new record...
Direct access after last parsed record (after line 956)
Jumped lines in file: 956
Found 956 already parsed records.
Parsed lines in file: 0
Found 0 dropped records,
Found 0 corrupted records,
Found 0 old records,
Found 0 new qualified records.
and so on

Site1 is the one giving me trouble

Annihilannic · August 4, 2008, 8:22pm

That doesn't look like logrotate output to me, it looks like the contents of the AWStats log files?

anderow · August 5, 2008, 5:12pm

Lol!

I know! Thats part of the problem....
That was:
[me@server1]# logrotate -f /etc/logrotate.conf

This is my /etc/logrotate.conf

Anyone know what awstats has done to my logrotate?!!!!!thanks

Ikon · August 5, 2008, 5:37pm

Do you have the files in /etc/logrotate.d for each of the virtual hosts logs location?

# cat /etc/logrotate.d/somevirtualhost
/var/www/vhosts/somevirtualhost.com/logs/*log {
    missingok
    notifempty
    sharedscripts
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}

Annihilannic · August 5, 2008, 7:27pm

Looking closer at your logrotate rule, it looks like the prerotate script, genstats.sh is the one generating that output.

What is the output of logrotate -v (you can chop out the genstats bit as we've alraedy seen it)?

anderow · August 8, 2008, 4:05pm

Thanks. Sorry for this slow reply.

Here is the output from the verbose command

[me@server ~]# logrotate -v /etc/logrotate.conf
reading config file /etc/logrotate.conf
including /etc/logrotate.d
reading config file acpid
reading config info for /var/log/acpid
reading config file cups
reading config info for /var/log/cups/_log
reading config file httpd
reading config info for /var/log/httpd/log
reading config file httpdvirtual
reading config info for /var/log/httpd//.accesslog /var/log/httpd//.errorlog /var/log/httpd//.sslerrorlog /var/log/httpd//.sslaccesslog
httpdvirtual: 4: the errors directive is deprecated and no longer used.
error: httpdvirtual:4 unexpected text
reading config file mgetty
reading config info for /var/log/mgetty.log.tty[^.] /var/log/mgetty.log.tty[^.][^.] /var/log/mgetty.log.tty[^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.unknown /var/log/mgetty.callback
reading config file ppp
reading config info for /var/log/ppp/connect-errors
reading config file psacct
reading config info for /var/account/pacct
reading config file rpm
reading config info for /var/log/rpmpkgs
reading config file syslog
reading config info for /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron
reading config file up2date
reading config info for /var/log/up2date
reading config file vsftpd.log
reading config info for /var/log/xferlog
reading config file yum
reading config info for /var/log/yum.log
reading config info for /var/log/wtmp
reading config info for /var/log/rkhunter.log

Handling 14 logs

rotating pattern: /var/log/acpid 65536 bytes (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/acpid
log does not need rotating

rotating pattern: /var/log/cups/*_log weekly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/cups/access_log
log does not need rotating
considering log /var/log/cups/error_log
log does not need rotating
not running shared postrotate script, since no logs were rotated

rotating pattern: /var/log/httpd/*log weekly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/httpd/access_log
log does not need rotating
considering log /var/log/httpd/error_log
log does not need rotating
considering log /var/log/httpd/ssl_access_log
log does not need rotating
considering log /var/log/httpd/ssl_error_log
log does not need rotating
considering log /var/log/httpd/ssl_request_log
log does not need rotating
not running shared prerotate script, since no logs will be rotated
not running shared postrotate script, since no logs were rotated

rotating pattern: /var/log/httpd//.accesslog /var/log/httpd//.errorlog /var/log/httpd//.sslerrorlog /var/log/httpd//.sslaccesslog weekly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/httpd//.accesslog
log /var/log/httpd//.accesslog does not exist -- skipping
considering log /var/log/httpd//.errorlog
log /var/log/httpd//.errorlog does not exist -- skipping
considering log /var/log/httpd//.sslerrorlog
log /var/log/httpd//.sslerrorlog does not exist -- skipping
considering log /var/log/httpd//.sslaccesslog
log /var/log/httpd//.sslaccesslog does not exist -- skipping
not running shared prerotate script, since no logs will be rotated
not running shared postrotate script, since no logs were rotated

rotating pattern: /var/log/mgetty.log.tty[^.] /var/log/mgetty.log.tty[^.][^.] /var/log/mgetty.log.tty[^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.][^.][^.] /var/log/mgetty.log.unknown /var/log/mgetty.callback weekly (4 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/mgetty.log.tty[^.]
log /var/log/mgetty.log.tty[^.] does not exist -- skipping
considering log /var/log/mgetty.log.tty[^.][^.]
log /var/log/mgetty.log.tty[^.][^.] does not exist -- skipping
considering log /var/log/mgetty.log.tty[^.][^.][^.]
log /var/log/mgetty.log.tty[^.][^.][^.] does not exist -- skipping
considering log /var/log/mgetty.log.tty[^.][^.][^.][^.]
log /var/log/mgetty.log.tty[^.][^.][^.][^.] does not exist -- skipping
considering log /var/log/mgetty.log.tty[^.][^.][^.][^.][^.]
log /var/log/mgetty.log.tty[^.][^.][^.][^.][^.] does not exist -- skipping
considering log /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.]
log /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.] does not exist -- skipping
considering log /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.]
log /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.] does not exist -- skipping
considering log /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.]
log /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.] does not exist -- skipping
considering log /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.][^.]
log /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.][^.] does not exist -- skipping
considering log /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.][^.][^.]
log /var/log/mgetty.log.tty[^.][^.][^.][^.][^.][^.][^.][^.][^.][^.] does not exist -- skipping
considering log /var/log/mgetty.log.unknown
log /var/log/mgetty.log.unknown does not exist -- skipping
considering log /var/log/mgetty.callback
log /var/log/mgetty.callback does not exist -- skipping

rotating pattern: /var/log/ppp/connect-errors after 1 days (5 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/ppp/connect-errors
log /var/log/ppp/connect-errors does not exist -- skipping

rotating pattern: /var/account/pacct after 1 days (31 rotations)
empty log files are not rotated, old logs are removed
considering log /var/account/pacct
log does not need rotating

rotating pattern: /var/log/rpmpkgs weekly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/rpmpkgs
log does not need rotating

rotating pattern: /var/log/messages /var/log/secure /var/log/maillog /var/log/spooler /var/log/boot.log /var/log/cron weekly (4 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/messages
log does not need rotating
considering log /var/log/secure
log does not need rotating
considering log /var/log/maillog
log does not need rotating
considering log /var/log/spooler
log does not need rotating
considering log /var/log/boot.log
log does not need rotating
considering log /var/log/cron
log does not need rotating
not running shared postrotate script, since no logs were rotated

rotating pattern: /var/log/up2date weekly (4 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/up2date
log does not need rotating

rotating pattern: /var/log/xferlog weekly (4 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/xferlog
log does not need rotating

rotating pattern: /var/log/yum.log 30720 bytes (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/yum.log
log does not need rotating

rotating pattern: /var/log/wtmp weekly (1 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/wtmp
log does not need rotating

rotating pattern: /var/log/rkhunter.log after 1 days (21 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/rkhunter.log
log does not need rotating

I dont understand why this

refuses to work!! it did before and now it doesnt!

My httpd log directories are

var/log/httpd/site1/site1.accesslog

I have tried again to force rotation and Ive still got the silly big log file?!

thanks for your help

Annihilannic · August 8, 2008, 6:26pm

It's because those patterns don't match those log file names. It needs to be /var/log/http/*/*.accesslog, for example.

anderow · August 8, 2008, 8:53pm

Thanks
Not sure how that happened....but I have corrected it and my site logs have now rotated

sometimes I cant see the wood for the trees

so thanks for the help mate..