I am using this command to remove all files and folders older than 24 hours;
find /databackup/dbs/* -maxdepth 0 -type d -mtime +0 -exec rm -rf {} \;
However, if/when backups to this folder fail and are not updated, it will completely destroy any backups we have in that folder.
What can I add to that line so it leaves the latest two folders (for example)
that were created.
So if backups don't run for a week I'm not stuck with an empty backup folder.
can someone help?
TIA
Post withdrawn.
I'm not very good at bash but I've been searching google for this and so far nothing without adding 10 lines of code and I still couldn't make heads or tails out of it.
Can anyone help?
So how is the backup folder structure, can we have the folder named as:
backup_20110122
backup_20110123
backup_20110124
backup_20110125
With that, you can list the folder name without the first 3 folders by sed '4,p' or awk 'NR>3'
Thanks for the reply.
the folder structure is like this;
180111
190111
200111
210111
240111
only dates.
is this how I would wrap it?:
find /databackup/dbs/* -maxdepth 0 -type d -mtime +0 | awk 'NR>3'
?
when I do this I only get 210111 showing up from the above folder list. Which means it displayed the folder that is 24+ hours old and ignored the oldest 3 folders from the listing.
without the NR>3 i get:
180111
190111
200111
210111
So it seems to just ignore the 3 oldest with the awk and only shows the latest one. (ignoring the previous/oldest 3)
I would want the script to ignore the latest 1,2 or 3 folders no matter what the time stamp is so it doesn't accidently remove all folders if backups don't run in 2 days.
Am I doing something wrong?
Your current date format is hard to sort, think to update in your backup script/cronjob and export folder format as below.
(I know there are some ways to resolve it, but can we have date format as: 110118 ?)
110118
110119
110120
110123
110125
find /databackup/dbs/* -maxdepth 0 -type d |sort -rn |awk 'NR>3'
... adding to rdcwayx...
find /databackup/dbs -maxdepth 0 -type d |sort -rn |awk 'NR>3' |
xargs rm -rf
I looked at your original post, and you said:
In the case you want to do it by date/time modification of the directory, you use:
ls -1tcp | grep "/$" | tail +3 | xargs echo rm -rf
When you are sure it's what you want, remove the "echo" from the xargs command.
Thanks for the help guys.
I will change the script so the date format is a little more standard.
When do i a ls -1tcp | grep "/$" | tail +3
I get the following error:
tail: cannot open `+3' for reading: No such file or directory
and when I change it to
ls -1tcp | grep "/$" | tail -3
the oldest 3 files folders get displayed.
this is what I need I think?
Then I will just xargs rm -rf ?
What does the grep "/$' do?
thanks.
I think you want ti delete all but the three newest. Try
tail -n +3
. If that does nit work, use
sed '1,3d'
.
The -p option in ls tells it to print directory names followed by a slash /. So the grep makes the distinction between file and directory. In case you had files in your backup folder.
The "-c" switch to "ls" is not the file creation timestamp. It is the timestamp the inode was last modified. Some backup software changes that timestamp. Be careful.
---------- Post updated at 17:25 ---------- Previous update was at 16:31 ----------
Here is an experimental script to display folders to delete but ensuring that we always have two folders left.
Script has an awkward sort on the directory names to sort ddmmyy to yymmdd order.
Assumes that directory only contains relevant folders.
# Exit if not at least 3 folders
counter=`ls -1d ?????? 2>/dev/null | wc -l`
if [ $counter -lt 3 ]
then
exit
fi
# Calculate how many folders to delete
counter2=`expr $counter - 2`
ls -1d ?????? | sort -k1.5n,1.6n -k1.3n,1.4n -k1.1n,1.2n | head -${counter2}
@methyl - That's a conundrum. The mtime will change anytime
A directory entry is modified.
Why would backup software touch the ctime? ( looking for a reasonable
Answer )
Commercial backup software would touch the "ctime" timestamp to mark the file as "backed up". Then use that timestamp as a reference for subsequent incremental backups. By using the unix filesystem this is vastly quicker than referencing an "all files" database. Not all backup packages do it.
Obviously they still need to refer to the "mtime" timestamp to find out what has changed. I can see a performance improvement on systems where the vast majority of changes are new files.
During the backup they then only need to update their own database with what has changed. In an incremental backup they must at some point detect deleted files.
IMHO. They are treating this timestamp as being similar to the archive flag on M$ platforms.
I have used the "ctime" timestamp to check that every file is actually backed up and uncovered some scary backup configurations.
Personally I object to backup software changing anything - especially when I invariably schedule a "full backup" not an "incremental" backup.
If nothing else it can be a PIA when diagnosing faults on a server where there are no proper records. I used to find out when a server was installed with "ls -lcd /".
Footnote: A backup can take several hours. An incremental backup needs to know exactly when the file was backed up. Personally I quiesce filesystems prior to a backup using every available technique to achieve this state.
thanks for all your help guys.
I managed it with the following:
change the format to yyyymmdd .
then I have this cronjob:
find /dbbackups/ -maxdepth 1 | egrep [0-9]{8} | sort -r | sed 1,3d | xargs rm -rf
ls -t would work but it dosn't output a full path so cron didn't know what to do with it.
find was the way to go output a full path so bash and/or cron knew where exactly the outputted folders were.
so even if backups will not occur, and as long as nothing else writes to that folder, it will find all the folders except the latest 3 and remove them.
I will be testing this for the next few days.
find /dbbackups/ -maxdepth 1 -type d -name "20[0-9][0-9]*" | sort -r | sed 1,3d | xargs rm -rf
So if anyone do a touch toto or a gzip logfile or does any other unexpected command for any bad or good reason into your backup directory, the inode may be modified doesn't it ?
I think i would also go for a better directory naming convention jut like YYYYMMDD and would then delete stuff based on a simple sorting, it seems to me more bullet proof as well as easier to troubleshoot if a backup directory disappear for any reason.