Logrotate and split file question

Hi,

We want to use logrotate to rotate the logfile but due to the size of the logfile.
We want to split the file first and then truncate -s the original file to a manageable size.

Using a sample file blah.log

$ du -sh blah.log
1.0G blah.log

Then we run

split -C 100M blah.log blah.log. -d

and it produces the following files?

blah.log.00
blah.log.01
blah.log.02
blah.log.03
blah.log.04
blah.log.05
blah.log.06
blah.log.07
blah.log.08
blah.log.09
blah.log.10

PROBLEM 1:
Is there a way to let it start as 1,2,3 ... i.e. blah.log.l, blah.log.2, i.e without the leading zero and start with 1 instead of 0?

PROBLEM 2:
I am hoping that blah.log.00 contains the latest entries and blah.log.10, the oldest entries
and unfortunately, that is not the case.

Because we will be using logrotate, we need the oldest entries on blah.log.10 and the earliest ones on blah.log.00
since logrotate has the older files in the high number, isn't it?

I can't find an option for split to start with blah.log.10, then blah.log.9 etc. so I think the only
option I have is to do it manually Or maybe someone else here have a trick that also does this? Logically, I suppose split has to start from low to high since it doesn't know how many files to split it to.

Both problems can be resolved manually or 'scripted' I believe, maybe someone in this group has done so in the past?

At the moment, I am working on renaming the splitted files using a script like below

#!/bin/bash
n=$( ls -1 blah.log.* | wc -l | awk '{ print $1 }' )
for file in $( ls -1 blah.log.* )
do
echo "mv $file blah.log.${n}"
n=$(( n - 1 ))
done

And then I just truncate -s the original blah.log file to a certain size.

BTW, while logrotate has copytruncate, I don't see an option for it to do truncate to a specific size, is that correct?
SO, copytruncate will always truncate the log file to zero size?

Any advise will be much appreciated. Thanks in advance.

Oh cr4p, looks like there's a huge bug in my script that it overwrites a file that has yet to be mv'ed

mv blah.log.00 blah.log.11
mv blah.log.01 blah.log.10 -- yikes ??
mv blah.log.02 blah.log.9
mv blah.log.03 blah.log.8
mv blah.log.04 blah.log.7
mv blah.log.05 blah.log.6
mv blah.log.06 blah.log.5
mv blah.log.07 blah.log.4
mv blah.log.08 blah.log.3
mv blah.log.09 blah.log.2
mv blah.log.10 blah.log.1 -- yikes ??

Use temporary file names to avoid name conflicts!

split -d -C 100M blah.log blah.tmp.log.
n=$(
  set -- blah.tmp.log.*
  echo $#
)
for file in blah.tmp.log.*
do
  echo mv "$file" bla.log.$n
  n=$(( n - 1 ))
done

Thanks, I will give this a try

Several problems here.

And then I just truncate -s the original blah.log file to a certain size.

Lines will always be appended to a log, so the oldest logs will stay in blah.log forever, and you will truncate all the later lines every time. You probably want to either recreate the log as empty, or copy some part of the end of the log to the front of a new file.

And in fact, those recent lines would almost certainly get duplicated in the next cycle of splitting, and probably several times depending on how many lines you retain. That is why logrotate truncates all lines: there is otherwise no way to avoid duplications.

Having two sets of files with the same suffix is not necessary. When you split the file, use the default alphabetic suffixing. Then count the files (e.g.10), and count in opposite directions renaming .a to .10, .b to .9 etc until you run out of numbers after .1.

The obvious issue there is that you can only renumber the files from .1 once: next time you run this, you will get duplicate filenames.

If you don't want 1GB files, why not configure logrotate to create 100MB files in the first place? They don't have to rotate by time -- IIRC you can trigger rotation by size too. If you want a date-time index, you could make that easily using the first and last date/times if the data contains those, or using the file metadata (which should be when the last line was written). Or you could simply rename each smaller log file to include a date/time, so you can search for the file(s) containing logs for a specific period directly.

Yes, you must truncate to zero size.
In the shell(-script) you can do it with a builtin command and a redirection:

: > blah.log

If you have logrotate, then its copytruncate does it. And yes, you can run it daily and act if the file size exceeds 100M.

I have used such a script once, to split up a yet not logrotated file. And then added it to the logrotate.

Hi,

Thanks for mentioning "so the oldest logs will stay in blah.log forever, and you will truncate all the later lines every time"

We thought the truncate will start from the latest line from the bottom up. I just did a test on that and you're right. Luckily, we are still in the testing phase.

The log file at the moment is 16G, we want to keep the latest 1G latest lines of the files and start logrotate from that. So we've truncate the 16G file to 1G and run the split on the 1G. And now, we know we're seriously wrong thinking we are keeping the latest lines

Just found out that we can use tail -c 1G blah16.log> blah.log. So now, we're just looking at using that and run the split on blah.log and then do > blah16.log to blank out the original log file and then run split on blah.log and rename it to the expected compressed filename that logrotate expects which are blah16.log.1.gz, blah16.log.2.gz etc

There is a subtlety in logrotate which you may not have picked up on yet.

The .1 file is always the most recently rotated log. When rotation happens, the entire list of logs is renamed, starting with the highest number. Like 5 ->6, 4->5, 3->4, 2->3, 1->2, in that order. Finally the new log is created as .1. If you have set a limit on the number of retained logs, then any files with a higher suffix than the limit are deleted.

If you number the logs yourself without conforming with that method, then unexpected deletions may happen.