Move multiple files 4rm Source to different target folders based on a series num in the file content

Dear Experts

my scenario is as follows...

I have one source folder "Source" and 2 target folders "Target_123456" & "Target_789101". I have 2 series of files. 123456 series and 789101 series. Each series has got 3 types of fiels "Debit", "Refund", "Claims".

All files are getting created in Source folder. I request for a unix script to open the file and check for its series (wheter its 123456 or 789101 series) number in its header and move the files to its corresponding Target folders "Target_123456" OR "Target_789101".

Please Note:
for DEBIT.DAT file the series number is available in line 1 i.e �VOL1000001 123456 �
for Claim & Refund files the series number is available in line 1 i.e �PWRD002123456 123456 00�

123456 Series files
DEBIT.DAT :-

VOL1000001                               123456                                1 
HDR1A123456S  112345600000100010001       16194 162000000000      
HDR2F0200000100                                   00    

CLAIM.DAT:-

PWRD001123456               123456 00  
VOL1000001                               123456                                1
HDR1A123456S  112345600000100010001       16295 162960000000
HDR2F0200000100                                   00                     

REFUND.DAT:-

PWRD002123456               123456 00
VOL1000002                               123456                                1
HDR1A      S  112345600000200010001       16294 162960000000
HDR2F0200000100                                   00     

789101 Series files
DEBIT.DAT:-

VOL1000001                               789101                                1
HDR1A789101S  178910100000100010001       16194 162000000000  
HDR2F0200000100                                   00     

CLAIM.DAT:-

PWRD001789101               789101 00
VOL1000001                               789101                                1
HDR1A789101S  178910100000100010001       16295 162960000000
HDR2F0200000100                                   00                     

REFUND.DAT:-

PWRD002789101               789101 00
VOL1000002                               789101                                1
HDR1A      S  178910100000200010001       16294 162960000000
HDR2F0200000100                                   00    

Any attempts/ideas/thoughts from your side?

1 Like

Hi

I am able to achieve this in Java. I am not comfortable in unix scripting. Tried searching for similar threads.

What be the target file names? Duplicates won't be allowed, neither in source nor in target directories.

Thanks a lot for your response.
Target file names should be same as source file names. duplicates should not be allowed.

SO - latest if each set has been moved once, duplicates will occur with the next action. Which measures do you intend to take for these cases?

I get files in source folder with unique names appended with ddmmyyyyhhmmss. e.g Debit25112016222930.dat

So no need to consider duplicate file scenario.

Upper case, lower case, or mixed case file names? However, try

for FN in *.dat; do read X SERIES X <"$FN"; echo mv "$FN" "Target_$SERIES/$FN"; done
mv Claim25112016222930.dat Target_789101/Claim25112016222930.dat
mv Debit25112016222930.dat Target_789101/Debit25112016222930.dat
mv Refund25112016222930.dat Target_789101/Refund25112016222930.dat

Remove the echo command if happy with the result.

1 Like

Hi RudiC

I am new to unix and not sure how to execute this. My requirement is to

  1. Script should check for .dat files in Source folder.
  2. Check for specific text (645559 or 508659) in first line of each file.
  3. If 645559 exists, move the files to folder SUN_ID_645559
  4. If 508659 exists, move the files to folder SUN_ID_508659

I am able to achieve this using java for windows platform. In jave we can hard code source and target directories. Write if conditons to search for the number in the first line of a file and if exists we can call a method to move this file to its corresponding folder.

I am trying to achive the same using unix as the requirement is to move the files in Unix and prefered solution is to use unix script. Not sure how to set variables for source and target folders? If condition for check for specific text in the first line and move the file to specific folders accordingly.

  package com.test;
  
 import java.io.BufferedReader;
 import java.io.BufferedWriter;
 import java.io.File;
 import java.io.FileNotFoundException;
 import java.io.FileReader;
 import java.io.FileWriter;
 import java.io.FilenameFilter;
 import java.io.IOException;
  
 public class MoveSunIDFiles {
  
     /**
      * @param args
      * @throws IOException
      */
     public static void main(String[] args)  {
         // TODO Auto-generated method stub
  
         File src = new File("C:\\Users\\akellap\\RND\\Test\\Source");
  
         // create new filename filter for .dat files
  
         File[] result = src.listFiles(new FilenameFilter() {
             public boolean accept(File dir, String name) {
                 if (name.lastIndexOf('.') > 0) {
                     // get last index for '.' char
                     int lastIndex = name.lastIndexOf('.');
  
                     // get extension
                     String str = name.substring(lastIndex);
  
                     // match path name extension
                     if (str.equals(".dat")) {
                         return true;
                     }
                 }
                 return false;
             }
         });
  
         if (result != null && result.length > 0) {
             for (int i = 0; i < result.length; i++) {
                 BufferedReader in = null;
                 try {
                     in = new BufferedReader(
                             new FileReader(result));
                 } catch (FileNotFoundException e) {
                     // TODO Auto-generated catch block
                     e.printStackTrace();
                 }
  
                 String line = "";
  
                 try {
                     while ((line = in.readLine()) != null) {
                         if (line.contains("645559")) {
                             move645559(result);
                             break;
                         }
  
                         if (line.contains("508659")) {
                             move508659(result);
                             break;
                         }
                     }
                 } catch (IOException e) {
                     // TODO Auto-generated catch block
                     e.printStackTrace();
                 }
  
             }
         } else if (result == null) {
             System.out.println("Invalid filename or folder");
         } else {
             System.out.println("No files matching the criteria");
         }
  
     }
  
     private static void move508659(File result) {
         // TODO Auto-generated method stub
  
         if (result.renameTo(new File(
                 "C:\\Users\\akellap\\RND\\Test\\Target\\SUN_ID_508659\\"
                         + result.getName()))) {
             System.out.println("File:" + result + " moved successfully");
         } else {
             System.out.println("File " + result.getName() + " not moved");
         }
     }
  
     private static void move645559(File result) {
         // TODO Auto-generated method stub
         if (result.renameTo(new File(
                 "C:\\Users\\akellap\\RND\\Test\\Target\\SUN_ID_645559\\"
                         + result.getName()))) {
             System.out.println("File:" + result + " moved successfully");
         } else {
             System.out.println("File " + result.getName() + " not moved");
         }
     }
 }

 

Sorry?
Where and how does my proposal not fulfill your requirement in post#1?

Hi

I have copied below lines of code into a notepad and saved as Script.sh. Placed the file in /Source folder.

for FN in *.dat; do read X SERIES X <"$FN"; echo mv "$FN" "SUN_ID_$SERIES/$FN"; done
mv Claim25112016222930.dat SUN_ID_789101/Claim25112016222930.dat
mv Debit25112016222930.dat SUN_ID_789101/Debit25112016222930.dat
mv Refund25112016222930.dat SUN_ID_789101/Refund25112016222930.dat

Placed around 100 .dat files in Source folder. Executed the script. I got below error.

-bash: Script.sh: command not found

Does below code means this script will handle only these 3 files?

mv Claim25112016222930.dat SUN_ID_789101/Claim25112016222930.dat
mv Debit25112016222930.dat SUN_ID_789101/Debit25112016222930.dat
mv Refund25112016222930.dat SUN_ID_789101/Refund25112016222930.dat

Make sure the script does not contain any DOS <CR> (^M, 0x0D, \r) line terminators.

And, the mv ... lines were the result of the one line script - remove them.

Hi RudiC

Thank You. Does this mean below code in unix


for FN in *.dat; do read X SERIES X <"$FN";
mv "$FN" "SUN_ID_$SERIES/$FN";
done

Will check for .dat files in Source folder.
Check for specific text (645559 or 508659) in first line of each file.
If 645559 exists, move the files to folder SUN_ID_645559
If 508659 exists, move the files to folder SUN_ID_508659

Yes - if "specific text (645559 or 508659)" is in the second white space separated field of the first line of the respective files.

That's why I added the echo - it shows what would happen without doing any possible harm.

Hi RudiC

Thanks a lot. This worked like a gem. It took lot of code to achieve in Java. Unix did it in 3 lines.

I am adding one line for logging purpose.

echo "$FN" moved successfully to target" "$(date)" >>log.txt

If possible can you please suggest some exception handling cased.

Assume if the source file does not have a Series ID, if the source file failed to move to other location etc...

Thanks again for your help.

You can look into mv 's -v (verbose) option to log what is actually done.
While I could post some error handling for this very basic snippet, I'd suggest you play around with e.g. if [ "$SERIES" -ne xxxxxx ] or if [ $? -ne 0 ] as an exercise to improve your shell scripting.

1 Like

Thanks a lot...I am exploring these options.

I am receiving files in below format(.DAT followed by 4 numbers). I the filter to *.DAT* or *.DAT???? . files are not getting picked tried searching for this in forums. No luck.

Sample1.DA9847
Sample2.DAT3498
Sample3.DAT9867

Observed one thing. This script is also picking the files which are getting copied along with existing files.

e.g. File A is existing and File B is getting copied (30% copied). When executed this script is moving File A (100%) and file B (30% copied) to target location. Is there anything to restrict in script to move only fully copied files?

Thanks in advance.

That is difficult to believe; I can't replicate this behaviour:

for FN in *.DAT????; do echo $FN; done
Sample2.DAT3498
Sample3.DAT9867

Of course, Sample1.DA9847 won't match *.DAT* nor *.DAT???? .

How do you tell copying is finished?