File extension search and copy

UNIX for Advanced & Expert Users
1

Hi need to know if we can write a shell script to find files for a particular format;s ie both .csv and .txt in a particular folder and then copy them to a new folder on a dialy basis. Does anyone know how this can be accomplished?

Thanks,
Sandeep

2

You can build a script around the command below and schedule it using some scheduler like cron.

find ./ -name "*.txt" -exec cp {} newfolder \;
3

Thanks. But need to to find both .txt and .csv files

Thanks,
Sandeep

4

Play with patterns like the one below.

"*.[ct][sx][vt]"

This will work if you don't have any file with extensions
.cst
.cxv
.cxt
.tsv
.tst
.txv

5

Can you please help me put this together in the find command and alos explain this a bit.

It will helpful.

Thanks,
Sandeep

6 
find ./ -name "*.[ct][sx][vt]" -exec cp {} newfolder \;

This command recursively finds from the current directory and subdirectories, files with names as per the below criteria.

After . the first character must be c or t (For .csv and .txt files)
After . the second character must be s or x (For .csv and .txt files)
After . the third character must be v or t (For .csv and .txt files)

Anyway, other combinations will also match (like .cxt) which I mentioned in my previous reply.

The exec in the above command copies all the found files (i.e, all .csv and .txt files we need) to the directory called newfolder.

Note: Better to have newfolder outside the current directory and not as a subdirectory in the present directory.

7

You can use
if [ -e ./OrigLocation/*csv ] && [ -e ./OrigLocation/*txt ]; then
.....
.....
else
....
....
fi

8

If they are all in the same folder then what's wrong with

cp folder/*.txt folder/*.csv otherlocation/

As far as that hideous find command line is concerned, is it really not more elegant to say

find . \( -name "*.csv" -o -name "*.txt" \) ...

(You have to escape the parens from the shell, something the find manual page doesn't typically stress enough.)

9

Thanks all. I had one lat thing i need to schedule it through cron to run every first wednesday of the month.

I know crontab -e will take me to cron and editing it in Vi editor:

50 23 * * 0-5 /usr/sandeep/TestAutomate.sh

how can we get this thing run every first wednesday of the month?

Thanks,
Sandeep

10

Add the following lines at the beginning of your script:

eval "date '+MM=%m'"
[ -f $HOME/TestAutomate.M$MM ] && exit
rm -f $HOME/TestAutomate.M??; touch $HOME/TestAutomate.M$MM

and schedule it to run on Wednesdays ONLY!!

11

can you please explain this a bit? Will be helpfull

Thanks,
Sandeep

12

Tthe traditional solution would be a self-scheduling at job. Then you can use arbitrary commands to specify the scheduling.

#!/bin/sh
: ... commands
when=`date -d "4 weeks"`
# code to figure out when to wait 5 weeks left as an exercise
echo "$0" "$@" | at $when
13

Unilover's solution relies on finding a file with this month's month number, and not doing the copying if it already exists. An alternative would be to compare the day number to 7, and only run if it is less. (Not having any external state would seem to me like an advantage.)

14

The first line saves the two-digit Current-Month number in the MM variable (e.g. MM=03 for March).

The second line checks to see if there is a file named AutoTester.M$MM (e.g. AutoTester.M03) in the user's Home-Directory, exits the script. So, for all subsequent Wednezdays (i.e. the second, third, ... ) in the month, it would exit without doing anything.

The third line, which will only be reached and executed when there is NO AutoTester.M$MM file (i.e. THE FIRST WEDNEZDAY OF EACH MONTH), first removes all previous such files (for previous months) and then creates that file for the CURRENT MONTH (which will cause the second line to sense it and exit for all subsequent executions on other Wednezdays) and then continues to the rest of the script.

15

I've just noticed that a pair of back-quotes are missing around the date-command in the first line. It should be:

eval "`date '+MM=%m'`"

Sorry!

16

where do you think i should add this part of code. should i run this external to the below code. Please let me know?

##!/usr/bin/sh -x

INPUT_DIR=/home/sadeep; export INPUT_DIR
OUTPUT_DIR=/usr/local/filefind/output; export OUTPUT_DIR
ERROR_DIR=/usr/local/filefind/error; export ERROR_DIR
LOG_DIR=/usr/local/filefind/log; export LOG_DIR

cd $INPUT_DIR

find $INPUT_DIR -name "*.[ct][sx][vt]"

if [ "$?" -ne 0 ]

then

    echo "No csv or txt files found in pwd" > $LOG_DIR/filetransfer.log



         exit 1

 else    

                 find $INPUT_DIR -name "*.[ct][sx][vt]" > filelist

          cat filelist >> $LOG_DIR/filetransfer.log

  fi

mkdir $OUTPUT_DIR/`date '+%m%d%y%`

           for nccfile $INPUT_DIR/*

       do

   echo "---------------- Moving files to output folder ---------------" >> $LOG_DIR/filetransfer.log
   echo "   " >> $LOG_DIR/filetransfer.log
   echo "   " >> $LOG_DIR/filetransfer.log

#read

            mv $nccfile $OUTPUT_DIR/\`date '\+%m%d%y%\`

            if [ "$?" -ne 0 ]

            then
                    echo $nccfile " - Move Failed......." >> $LOG_DIR/filetransfer.log
                           echo "   " >> $LOG_DIR/filetransfer.log
                           echo "   " >> $LOG_DIR/filetransfer.log
            else
                    echo $nccfile " - Moved Successfully to $OUTPUT_DIR" >> $LOG_DIR/filetransfer.log
                    echo "   " >> $LOG_DIR/filetransfer.log
  			echo "   " >> $LOG_DIR/filetransfer.log


                    
            fi
            
    done

exit 0

17

before

INPUT_DIR=/home/sadeep; export INPUT_DIR

18

thanks very much

19

Hi Bsandeep,

Put this entry in cron:

50 23 1-7 * * test `date +\%a` != Wed || /usr/sandeep/TestAutomate.sh

20

Or rather just schedule it to run on Wednesdays by putting a 3 in the fifth column as:

50 23 * * 3 /usr/sandeep/TestAutomate.sh