Find all files other than first two files dates & last file date for month

Makarand_Dodmis · April 10, 2014, 11:42am

Hi All,
I need to find all files other than first two files dates & last file date for month and month/year wise list.

lets say there are following files in directory

Mar 19 2012 c.txt
Mar 19 2012 cc.txt
Mar 21 2012 d.txt
Mar 22 2012 f.txt
Mar 24 2012 h.txt
Mar 25 2012 w.txt
 
Feb 12 2012 q.txt
Feb 21 2012 a.txt
Feb 21 2012 aa.txt
Feb 22 2012 s.txt
Feb 23 2012 k.txt
Feb 27 2012 j.txt
 
Mar 19 15:43 c.txt
Mar 21 15:43 d.txt
Mar 22 15:43 f.txt
Mar 24 15:43 h.txt
Mar 25 15:43 w.txt
Mar 25 15:43 ww.txt
 
Feb 12 15:43 q.txt
Feb 21 15:43 a.txt
Feb 22 15:43 s.txt
Feb 23 15:43 k.txt
Feb 27 15:43 j.txt
Feb 27 15:43 jj.txt

output should be:

for Mar 2012
Mar 22 2012 f.txt
Mar 24 2012 h.txt
 
for Feb 2012
Feb 22 2012 s.txt
Feb 23 2012 k.txt
 
for Mar 2014
Mar 22 15:43 f.txt
Mar 24 15:43 h.txt
 
for Feb 2014
Feb 22 15:43 s.txt
Feb 23 15:43 k.txt

RudiC · April 10, 2014, 1:26pm

Please use code tags as required by forum rules!

How about trying to adopt & adapt what you learned from your other tread?
And, your specification is not consistent with your sample output. Where are

Mar 21 2012 d.txt
Feb 21 2012 aa.txt
Mar 25 15:43 w.txt
Feb 27 15:43 j.txt

?

Makarand_Dodmis · April 11, 2014, 5:20am

I did not get what you not understand from above question.

I have clearly mention that first two file dates & last files dates files i dont want in the output list.

rest all files i want. thats it.

vbe · April 14, 2014, 9:35am

My way of seeing things is you know what you dont want : search for them first and create a list with them, then parse the directory excluding the list content

MONTH=Sep
-rwxr-xr-x    1 vbe      bin             110 Sep  3 2003  tutu.sav
-rwxr-x---    1 vbe      bin              17 Sep  4 2003  tata
-rw-rw-r--    1 vbe      system         7696 Sep 30 2013  bdf_today.130930
n12:/home/vbe $ ll -lrt|grep " Sep "|grep -v -f test.0009|more                                  
-rw-rw-rw-    1 vbe      bin              21 Sep  9 2003  testread.out
-rw-rw-rw-    1 vbe      bin            1940 Sep  9 2003  cutresults
-rwxrw-rw-    1 vbe      bin             237 Sep  9 2003  cuttest
-rw-rw-rw-    1 vbe      bin             101 Sep 17 2003  newterm
-rwxr-xr-x    1 vbe      bin             394 Sep 17 2003  whiletest
-rw-rw-rw-    1 vbe      bin            2920 Sep 17 2003  shell_cpt9
-rw-rw-rw-    1 vbe      bin             121 Sep 18 2003  sendmail_cf.modif
-rw-rw-rw-    1 vbe      bin             626 Sep 18 2003  lvreduce.howto
-rw-rw-rw-    1 vbe      bin            1063 Sep 18 2003  gagatest
-rwxr-xr-x    1 vbe      bin              80 Sep 26 2003  popu
-rwxr-xr-x    1 vbe      bin             718 Sep 30 2003  cpt9test
-rw-rw-rw-    1 vbe      bin              17 Sep 30 2003  errorlog
-rw-rw-rw-    1 vbe      bin             450 Sep 30 2003  usersname
-rw-rw-rw-    1 vbe      bin             128 Sep 14 2004  bufpages.howto
-rw-rw-rw-    1 vbe      bin             136 Sep 27 2004  lup.c
-rwxr-xr-x    1 vbe      bin              92 Sep 27 2004  sh_test3
-rwxr-xr-x    1 vbe      bin              13 Sep 27 2004  sh_test1
-rw-r-----    1 vbe      bin          163448 Sep 28 2004  titi.out
-rw-r--r--    1 vbe      bin          125670 Sep 28 2004  JAVAinfo.out
-rw-rw-rw-    1 vbe      bin             931 Sep 29 2004  mozilla_bug
-rw-rw-rw-    1 vbe      bin            1605 Sep  1 2005  GW_lan_discon.howto
-rw-r--r--    1 vbe      bin            1871 Sep  1 2005  sasperms.logs
-rw-rw-rw-    1 vbe      bin           25940 Sep  2 2005  sasproc_log.log
-rwxr-xr--    1 vbe      bin             321 Sep  7 2005  processor.sh
-rw-rw-rw-    1 vbe      bin              68 Sep  7 2005  node.dat
-rw-rw-rw-    1 vbe      bin             364 Sep  7 2005  node_name_script
-rw-r--r--    1 vbe      bin             136 Sep 12 2005  net_perf_w_glance.howto

vbe · April 14, 2014, 9:39am

So what you dont want: (create list...)

for i in $(ls -l |awk '{print $6}'|sort|uniq )
do
   echo MONTH=$i
   ls -lrt|grep ^- |grep " $i "|head -2|awk '{print $9}' >>test.0009
   ls -lrt|grep ^- |grep " $i "|tail -1|awk '{print $9}' >>test.0009
done

Then as previous post ( but thats an example...)

Makarand_Dodmis · April 14, 2014, 9:40am

Hi Vbe

I have working code with parsing the directory. but it is taking lot of time .
I want faster solution..Any help

vbe · April 14, 2014, 9:46am

Well based on what you said we could not guess the number of files impacted... I would go processing by month so you can do some in parallele ( that means also creating a exclusion list by month)

Makarand_Dodmis · April 14, 2014, 9:53am

Also the main problem with below command

ls -lrt|grep ^- |grep " $i "|head -2|awk '{print $9}'

is only giving last two files . i want all files having date same as date of first two files.
similarly i want all files having date same as date of last file of month.

this i want for all months.

for example
jun 2012
mar 2013
apr 2013
mar 2014
apr 2014

vbe · April 14, 2014, 10:03am

Post #1 you wrote:

As I said in my preious post the easiest to cope with is to make a list with the unwanted and use that with a grep -v
If you want months per year, you will have to deal year by year...

Makarand_Dodmis · April 14, 2014, 10:06am

i will use

grep -v

that's not a issue.

could you send me command for all files with first two files dates & last file date for month

Note :suppose there 4 files with 1st mar 2014 then i want all 4 not the only 1st file
Hope you got my requirement

vbe · April 14, 2014, 10:13am

Was in previous post:

for i in $(ls -l |awk '{print $6}'|sort|uniq ) # list what months we have in this directory
do
   echo MONTH=$i
   ls -lrt|grep ^- |grep " $i "|head -2|awk '{print $9}' >>test.0009 # 2 first of the month
   ls -lrt|grep ^- |grep " $i "|tail -1|awk '{print $9}' >>test.0009 #thelast for that month
done

Since this creates the exclusion list you would have to start by counting ow many files for that month and if more than 4 then create reacord in exclusion list

vbe · April 14, 2014, 10:22am

So the code to produce exclusion list could look like:

for i in $(ls -l |awk '{print $6}'|sort|uniq )
do
   MIN=$(ls -l |grep " $i "|wc -l)
   echo MONTH=$i " MIN="$MIN
   if [ $MIN -gt 4 ]
   then
      ls -lrt|grep ^- |grep " $i "|head -2|awk '{print $9}' >>test.0009
      ls -lrt|grep ^- |grep " $i "|tail -1|awk '{print $9}' >>test.0009
   fi
done

Makarand_Dodmis · April 14, 2014, 10:32am

i think unfortunatly we both are not same page

1) This count 4 is not constant
2) for e.g. If there are two files with date 01/03/2014
3) for e.g. If there are three files with date 02/03/2014
4) for e.g. If there are six with date 15/03/2014
5) for e.g. If there are five files with date 31/03/2014

6) then i want files from above point 2),3)&5) (01/03/2014,02/03/2014,31/03/2014)
total 2+3+5= 10 files

7) the 6 files with date 15/03/2014 should not be there

so either give me list with above 10 files point #6

or

list with 6 files point #7

vbe · April 14, 2014, 11:29am

Perhaps if you showed us your code, we could see if there is a faster way of doing...

Makarand_Dodmis · April 14, 2014, 11:55am

XXXDIR=/abc/def/ghi
#not months first two & last one
function file_purge
{
    tradecheck=`pwd`
 echo "Files to purge...."
    $1| while read dm_file
    do   
  day=`perl -MPOSIX -le 'print strftime "%d", localtime((lstat)[9]) for @ARGV' "$dm_file"`
  mon=`perl -MPOSIX -le 'print strftime "%b", localtime((lstat)[9]) for @ARGV' "$dm_file"`
        year=`perl -MPOSIX -le 'print strftime "%Y ", localtime((lstat)[9]) for @ARGV' "$dm_file"`
          dday=`expr $day + 0`
    
    cntr=0
    cntr2=0
    flag1=0
    flag2=0
    flag3=0
        #This is list of files for same month & year of $dm_file for more than 180 days old
        ls -ltr * |nawk -v mon=$mon -v year=$year '{if ($8 == year && $6 == mon) {print $9}}'| while read inner_file
         do
            imfile=`find $tradecheck -name $inner_file`
            i_day=`perl -MPOSIX -le 'print strftime "%d", localtime((lstat)[9]) for @ARGV' "$imfile"`
            i_dday=`expr $i_day + 0`
            purge_first "$dday" "$i_dday"
            purge_last "$dday" "$i_dday"
         done
         
        if [ $flag1 -eq 0 ] && [ $flag2 -eq 0 ] && [ $flag3 -eq 0 ]; then
         echo $dm_file $day $mon $year    
        fi
  
 done
}
#For confirming file not the last file's date
function purge_last
{
if [ $1 -ge $2 ]; then
 flag3=1
else
 flag3=0
 break    
fi
}
#For confirming file not the first two file's date
function purge_first
{
if [ $1 -le $2 ]; then
 flag1=1
else
   
 if [ $cntr2 -eq 0 ]; then
  check_day=$2
  cntr2=`expr $cntr2 + 1`
 fi
 
 if [ ! $check_day -eq $2 ]; then
  cntr=`expr $cntr + 1`
 fi
fi
if [ $cntr -eq 0 ]; then
 flag2=1
else
 flag1=0
 flag2=0
fi
}
  
# purge the files which are 180 days older
cd $XXXDIR
var3="find . ! -name -prune -type f -mtime +180"
file_purge "$var3"

---------- Post updated at 10:43 AM ---------- Previous update was at 10:42 AM ----------

i want to find files for 6 months old.hence passing that to function file_purge

---------- Post updated at 10:55 AM ---------- Previous update was at 10:43 AM ----------

XXXDIR=/abc/def/ghi
#not months first two & last one
function file_purge
{
    tradecheck=`pwd`
 echo "Files to purge...."
    $1| while read dm_file
    do   
  day=`perl -MPOSIX -le 'print strftime "%d", localtime((lstat)[9]) for @ARGV' "$dm_file"`
  mon=`perl -MPOSIX -le 'print strftime "%b", localtime((lstat)[9]) for @ARGV' "$dm_file"`
        year=`perl -MPOSIX -le 'print strftime "%Y ", localtime((lstat)[9]) for @ARGV' "$dm_file"`
          dday=`expr $day + 0`
    
    cntr=0
    cntr2=0
    flag1=0
    flag2=0
    flag3=0
        #This is list of files for same month & year of $dm_file for more than 180 days old
        ls -ltr * |nawk -v mon=$mon -v year=$year '{if ($8 == year && $6 == mon) {print $9}}'| while read inner_file
         do
            imfile=`find $tradecheck -name $inner_file`
            i_day=`perl -MPOSIX -le 'print strftime "%d", localtime((lstat)[9]) for @ARGV' "$imfile"`
            i_dday=`expr $i_day + 0`
            purge_first "$dday" "$i_dday"
            purge_last "$dday" "$i_dday"
         done
         
        if [ $flag1 -eq 0 ] && [ $flag2 -eq 0 ] && [ $flag3 -eq 0 ]; then
         echo $dm_file $day $mon $year    
        fi
  
 done
}
#For confirming file not the last file's date
function purge_last
{
if [ $1 -ge $2 ]; then
 flag3=1
else
 flag3=0
 break    
fi
}
#For confirming file not the first two file's date
function purge_first
{
if [ $1 -le $2 ]; then
 flag1=1
else
   
 if [ $cntr2 -eq 0 ]; then
  check_day=$2
  cntr2=`expr $cntr2 + 1`
 fi
 
 if [ ! $check_day -eq $2 ]; then
  cntr=`expr $cntr + 1`
 fi
fi
if [ $cntr -eq 0 ]; then
 flag2=1
else
 flag1=0
 flag2=0
fi
}
  
# purge the files which are 180 days older
cd $XXXDIR
var3="find . ! -name -prune -type f -mtime +180"
file_purge "$var3"

vbe · April 14, 2014, 12:31pm

why dont you start with eliminating the old files more than 180 days?
I understand after all files to be processed will be less than 3 months, is that so? ( makes a change on posiible 12 months x X years...)

Makarand_Dodmis · April 15, 2014, 4:21am

I dont want to delete all files > 180Requirement is thispurge files > 180 days and file date should not be first two files dates & last file date for month