Logrotate and Compressing only yesterdays files

Hello,

I have a syslog server at home and am currently experiencing an issue where my logs will rotate and compress however it will rotate and compress yesterdays file and the newly created log file for the current day. When it does this however it will also create another new file for today and as a result my compressed files look like this:

4.8M Aug 12 23:59 router.2019.08.12.log.1.gz--> this will be the encrypted log file from the 11th of August
5.3K Aug 12 00:02 router.2019.08.12.log.2.gz--> this will be the encrypted log file from the 12th of August Morning
37M Aug 12 00:02 router.2019.08.12.log --> this will be the new current day log file

Here is a copy of my syslog-ng.conf:

/var/log/router/router.*.log
{
        rotate 2
        daily
        missingok
        notifempty
        postrotate
                invoke-rc.d syslog-ng reload > /dev/null
        endscript
}

Here is the section in my syslog-ng.conf file that is relevant to the logs above

# Router ###########
# Add Filter to add our router
filter router { host( "192.168.XX.1" ); };
# Add destination file where logs will be stored
#destination df_router { file("/home/pi/files/router.log"); };
log { source ( s_net ); filter( f_router ); destination ( df_router ); };
destination df_router {
file("/var/log/router/router.${YEAR}.${MONTH}.${DAY}.log"
template-escape(no));
};

Here is my cron job that is running:

02 00 * * * sudo logrotate --force /etc/logrotate.d/syslog-ng

My question is this, what do I do in order to make sure that only yesterdays file gets compressed at 2 minutes beyond midnight instead of both current and yesterday files. If there is any other information required don't hesistate to ask.

Sorry, in your post, you mean "compressed file" and not "encrypted file" for your .gz files right?

Yes sorry I just updated it. For some reason I am always defaulting to encrypted.

Seems to me from youf post that the current day logfile is not compressed.

This is standard for logrotate , to compress the logs from the prior prior when logs are rotated.

Hi Neo,

What will happen is logrotate will create a new file for today when we hit midnight as it should.
However when the compression runs it compresses both the newly created .log file for the new day and the .log file from yesterday.
It will still however end up creating a second log file for the new day that starts where the initial one left off so in this case 2 minutes after midnight.
In the morning I end up with two compressed .log files and a none compressed .log file.
Here is a screenshot from windows perpsective. The compressed file at the top would be files from the 11th and the one at the bottom would be the first 2 minutes after midnight for the 12th.
What isn't shown is the third actual log file that would be there (I can't show current because I tried something that didn't work out).
Also apologies for the massive thumbnail.

Most distros have a default crontab entry for logrotate that runs all rule files in /etc/logrotate.d/
(Once a day. Check all the cron files /etc/crontab /etc/cron.d/* /etc/cron.daily/* for the default logrotate!)
With your dedicated crontab entry the rule file /etc/logrotate.d/syslog-ng will run twice.

Hello,

There is a default logrotate in /etc/daily which looks like this

#!/bin/sh

test -x /usr/sbin/logrotate || exit 0
/usr/sbin/logrotate /etc/logrotate.conf

I looked in the logrotate.conf file and have this and just now uncommented compress and create.

# see "man logrotate" for details
# rotate log files weekly
daily

# keep 4 weeks worth of backlogs
rotate 2

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

I removed my entries in crontab that were forcing the rotation to run using the syslog-ng.conf file.
I shall see what happens tomorrow/tonight and will let you know. Thanks to you both for the help in the mean time!

Hello,

So I don't know if it's a win or not but on the bright side it didn't compress more than once on the down side it didn't compress at all.
Wondering if I should move the file to a new folder and compress it there instead.

FYI:

server:/etc# cd logrotate.d
server:/etc/logrotate.d# ls -l
total 44
-rw-r--r-- 1 root root 120 Nov  2  2017 alternatives
-rw-r--r-- 1 root root 452 Jul 31 08:00 apache2
-rw-r--r-- 1 root root 173 Apr 20  2018 apt
-rw-r--r-- 1 root root  79 Jan 16  2018 aptitude
-rw-r--r-- 1 root root  82 Jul 21  2018 certbot
-rw-r--r-- 1 root root 107 Sep 20  2016 dbconfig-common
-rw-r--r-- 1 root root 112 Nov  2  2017 dpkg
-rw-r--r-- 1 root root 846 Jul 31 08:00 mysql-server
-rw-r--r-- 1 root root 501 Jan 14  2018 rsyslog
-rw-r--r-- 1 root root 178 Aug 15  2017 ufw
-rw-r--r-- 1 root root 235 Jul 18  2018 unattended-upgrades
server:/etc/logrotate.d# cat rsyslog
/var/log/syslog
{
	rotate 7
	daily
	missingok
	notifempty
	delaycompress
	compress
	postrotate
		/usr/lib/rsyslog/rsyslog-rotate
	endscript
}

/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
	rotate 4
	weekly
	missingok
	notifempty
	compress
	delaycompress
	sharedscripts
	postrotate
		/usr/lib/rsyslog/rsyslog-rotate
	endscript
}
server:/etc/logrotate.d# 

Hope this helps.

delaycompress will not compress the first backup, but start compressing with the second backup if compress is used.
If globally (in /etc/logrotate.conf) there is delaycompress, then it can be inhibited in an individual rule file with nodelaycompress.
Some daemons do not like compression on their current/open log file(s), then delaycompress or copytruncate is necessary.