Conditional delete

Hi RudiC

here the modified version and output:

[asp@abcsd34 ~]$ cat abc.sh
{ cd /home/asp; stat -fc"%b %a %S" .; stat -c"%n %b %B" 2018*.tar.gz | sort; } | awk '
NR == 1         {print Needed = ($1 * PCT - $2) * $3
                 next
                }

                {split ($1, T, "_")
                 if (T[1] == KnownTime)
                 KnownTime = T[1]
                 LastFile = $1
                 {print "echo rm " LastFile
                   print SUM += $2 * $3
                   if (SUM >= Needed) exit
                  }
                 }
' PCT=0.4
[asp@abcsd34 ~]$ sh abc.sh
5.63412e+09
echo rm 20180701_0012.tar.gz
251428864
echo rm 20180701_0712.tar.gz
502857728
echo rm 20180701_1412.tar.gz
754286592
echo rm 20180701_2112.tar.gz
1005715456
echo rm 20180702_0012.tar.gz
1257144320
echo rm 20180702_0712.tar.gz
1508573184
echo rm 20180702_1412.tar.gz
1760002048
echo rm 20180702_2112.tar.gz
2011430912
echo rm 20180703_0012.tar.gz
2262859776
echo rm 20180703_0712.tar.gz
2514288640
echo rm 20180703_1412.tar.gz
2765717504
echo rm 20180703_2112.tar.gz
3017146368
echo rm 20180704_0012.tar.gz
3268575232
echo rm 20180704_1412.tar.gz
3520004096
echo rm 20180704_2112.tar.gz
3771432960
echo rm 20180705_0012.tar.gz
4022861824
echo rm 20180705_1412.tar.gz
4274290688
echo rm 20180705_2112.tar.gz
4525719552
echo rm 20180706_0012.tar.gz
4777148416
echo rm 20180706_1412.tar.gz
5028577280
echo rm 20180706_2112.tar.gz
5280006144
echo rm 20180707_0012.tar.gz
5531435008
echo rm 20180707_1412.tar.gz
5782863872
[asp@abcsd34 ~]$

But if you see output, it showing all files for given day..its not leaving last copy of the day...

OK, my fault, I may have taken too much for granted. Pls. modify as

                 if (T[1] == KnownTime)         {print "echo rm " LastFile
                                                 print SUM += $2 * $3
                                                 if (SUM >= Needed) exit
                                                }
                 KnownTime = T[1]
                 LastFile = $1
                }

First of thank you very much for all you patience with me
this thread become hottest thread..

Here the modified the code

[asp@abcsd34 ~]$ cat abc7.sh
{ cd /home/asp; stat -fc"%b %a %S" .; stat -c"%n %b %B" 2018*.tar.gz | sort; } | awk '
NR == 1         {print Needed = ($1 * PCT - $2) * $3
                 next
                }

                {split ($1, T, "_")
                 if (T[1] == KnownTime)
                                   {print "echo rm " LastFile
                    print SUM += $2 * $3
                    if (SUM >= Needed) exit
                    }
                 KnownTime = T[1]
                 LastFile = $1
                }
' PCT=0.4
[asp@abcsd34 ~]$ sh abc7.sh
5.63364e+09
echo rm 20180701_0012.tar.gz
251428864
echo rm 20180701_0712.tar.gz
502857728
echo rm 20180701_1412.tar.gz
754286592
echo rm 20180702_0012.tar.gz
1005715456
echo rm 20180702_0712.tar.gz
1257144320
echo rm 20180702_1412.tar.gz
1508573184
echo rm 20180703_0012.tar.gz
1760002048
echo rm 20180703_0712.tar.gz
2011430912
echo rm 20180703_1412.tar.gz
2262859776
echo rm 20180704_0012.tar.gz
2514288640
echo rm 20180704_1412.tar.gz
2765717504
echo rm 20180705_0012.tar.gz
3017146368
echo rm 20180705_1412.tar.gz
3268575232
echo rm 20180706_0012.tar.gz
3520004096
echo rm 20180706_1412.tar.gz
3771432960
echo rm 20180707_0012.tar.gz
4022861824
echo rm 20180707_1412.tar.gz
4274290688
echo rm 20180708_0012.tar.gz
4525719552
echo rm 20180708_1412.tar.gz
4777148416
echo rm 20180709_0012.tar.gz
5028577280
echo rm 20180709_1412.tar.gz
5280006144
echo rm 20180710_0012.tar.gz
5531435008
echo rm 20180710_1412.tar.gz
5782863872
[asp@abcsd34 ~]$

Now its showing correctly first 3 files of the day in echo and skipping one which needs to be retained.

Looks like your problem is solved? If you REALLY trust it, remove echo and pipe through sh to make it fully functional.

As Peasant in post#16 proposed, you may want to increase the compression factor of your backup gzip to save some space?

Sorry at my poor knowledge...did not get you mean by pipe through sh...
can you give small example please...

" If you Really trust" --haha no other go, i need to test it for sure whether it will come correct or not.. A Big thank you for your help

I'm confused about a couple of points...
1st: In the pipeline feeding into awk :

stat -c"%n %b %B" 2018*.tar.gz | sort; 

why is the sort needed? The shell will expand 2018*.tar.gz in increasing sorted order by the date and time stamps included in the filenames matched by that pattern and sort should not change that order.

2nd: In post #8, it was pointed out that the filename matching pattern I suggested in post #4 has a Y3K problem (i.e., would stop working in January, 3000). Using the filename matching pattern shown above (i.e., 2018*.tar.gz , the code will stop working in January, 2019. Do you really want to use a pattern that only matches files created this year?

regarding 2 nd point, obviously I need this to work after 2018 as well .

Did you read and understand Don Cragun's post#4?

Understood - I wasn't sure stat 's output would be sorted although the shell expands / presents the file names sorted.

The proposal (as mainly all of my proposals) is to be read as a "proof of concept" and not as a "turnkey system". It does NOT relieve the user of developing a deep understanding and thoroughly checking / testing / adapting it before applying to real world - even less to production - systems. Please adapt to your needs, e.g. by reading, understanding, and applying other posts in this thread.

Hi RudiC/Don Cragun ,

I tested it successfully the below code by piping to sh (thanks to Post no 4 and googled and understood it), its deleting right files and keeping latest one and deleting rest till its get the space to 60 % Thank you all who ever replied on this post..

{ cd /home/asp; stat -fc"%b %a %S" .; stat -c"%n %b %B" 2018*.tar.gz | sort; } | awk '
NR == 1         {Needed = ($1 * PCT - $2) * $3
                 next
                }

                {split ($1, T, "_")
                 if (T[1] == KnownTime)
                    {print "rm " LastFile
                    SUM += $2 * $3
                    if (SUM >= Needed) exit
                    }
                 KnownTime = T[1]
                 LastFile = $1
                }
' PCT=0.4  |sh

[asp@abcsd34 ~]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        40G   24G   17G  60% /
devtmpfs        1.9G     0  1.9G   0% /dev
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G   18M  1.9G   1% /run
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
tmpfs           377M     0  377M   0% /run/user/1000

I m considering to add additional functionality(is this correct way)

find /path/to -type f -mtime +30 -exec rm {} \;

as we don't need any file before 30 days from sysdate, however i could not test as sample files i have created have same timestamp(real timestamp not the fine name though)

Hi

it was working on some systems..But now on a particular machine it s saying cannot remove file no such file or directory...even though file is there. I am not sure if its complain about space in file name, but when i put only echo its showing file names...

{ cd /home/asp; stat -fc"%b %a %S" .; stat -c"%n %b %B" 2018*.tar.gz | sort; } | awk '
NR == 1         {Needed = ($1 * PCT - $2) * $3
                 next
                }

                {split ($1, T, "_")
                 if (T[1] == KnownTime)
                    {print "rm " LastFile
                    SUM += $2 * $3
                    if (SUM >= Needed) exit
                    }
                 KnownTime = T[1]
                 LastFile = $1
                }
' PCT=0.4  |sh