Dears,
I have a listfile contains list of files path.
i need to read the line of the listfile
mv the file to other directory
and update the listfile by deleting the lines of the listfile.
#!/bin/bash
target=/fstest/INVESTIG/Sadiq/TEST_ARCH
while read -r line || [[ -n $line ]];
do
mv $line $target
done < /fstest/INVESTIG/Sadiq/processedfile.txt
Regards,
Sadique
You need to take into account that the mv command can fail. So you need to keep track of failed files.
#!/bin/bash
target=/fstest/INVESTIG/Sadiq/TEST_ARCH
> /tmp/errfile # create an empty file
while read -r line || [[ -n $line ]];
do
mv "$line" $target || echo "$line" >> /tmp/errfile
done < /fstest/INVESTIG/Sadiq/processedfile.txt
mv /tmp/errfile /fstest/INVESTIG/Sadiq/processedfile.txt
# /fstest/INVESTIG/Sadiq/processedfile.txt will be empty if no mv failures occurred
# otherwise the names of files that were not moved still remain in the original input file
filenames with spaces may be a problem, too - why the " quotes surround the $line
We've already pointed out the pointlessness of the || [[ -n $line ]] construct - you might consider to get rid of it.
Does your mv version provide the -v option? Does your grep offer the -x option? Would, then, this come close to what you need?
while read line; do mv -v $line $target; done <file 2>/dev/null | sed 's/\o047\| ->.*$//g; s/^/^/' | grep -vxf- file
Hello RudiC,
As per your suggestion, i removed the [[-n $line ]] after the moved from source to target, I still find the those lines inside the filelist.
yes mv version provide the -v option.
so do grep offer the -x .
its Linux Redhat vesrsion 7.2
Kindly suggest.
Regards,
sadique
#!/bin/bash
target=/fstest/INVESTIG/Sadiq/TEST_ARCH
while read -r line # is added && [[ -n $line ]]; better?
do
mv -fb $line $target && echo $line>>/fstest/INVESTIG/Sadiq/processedfile_MOVED.txt || echo $line>>/fstest/INVESTIG/Sadiq/processedfile_NOTMOVED.txt
done < /fstest/INVESTIG/Sadiq/processedfile.txt
Dear Abdul,
My concern is that i want to update the listfile after each path(line) moved from it can be delete from the processedfile_.txt filelist.
Say:
processedfile_.txt filelist contains:
/fstest/INVESTIG/Sadiq/TEST/file001.bin
/fstest/INVESTIG/Sadiq/TEST/file002.bin
/fstest/INVESTIG/Sadiq/TEST/file003.bin
/fstest/INVESTIG/Sadiq/TEST/file004.bin
/fstest/INVESTIG/Sadiq/TEST/file005.bin
/fstest/INVESTIG/Sadiq/TEST/file006.bin
/fstest/INVESTIG/Sadiq/TEST/file007.bin
/fstest/INVESTIG/Sadiq/TEST/file008.bin
/fstest/INVESTIG/Sadiq/TEST/file009.bin
/fstest/INVESTIG/Sadiq/TEST/file010.bin
when reading each line it shud move to target directory.
but after moved, script should able to update filelist by deleting the line of the path which has been moved.
The output of above IS the reduced file list. Capture it and mv to original file list.
Dear RudiC,
I am confused from your above post.
Kindly guide
What is the output of the proposal in post#3? Compare this to the original file. It should be the file names for which the mv failed.
what if my file list its keep on appending.
so i want to read each line perform move command and then delete the line from the listfiles one by one simultaneously.
Appending? No way - if you run the snippet as given. I've (reluctantly) built your sceanrio from post#6 but removed file003.bin from the input file. Result:
while read line; do cp -v $line /tmp/; done <file 2>/dev/null | sed 's/\o047\| ->.*$//g; s/^/^/' | grep -vxf- file
TEST/file003.bin
which - to me - is EXACTLY what you requested : the input file with ALL successful moves deleted. Save this result to a file and replace the input
You could also try - given the input is sorted -
while read line; do cp -v $line /tmp/; done <file 2>/dev/null | cut -d\' -f2 | comm -13 - file
Sorry for the confusion.
what i mean to say is my input filelist is keep appending through a program.
What i need to do is move this files whose path has be captured in the filelist to another directory.
and once moved delete line from the filelist not file from the directory
You say another process has the input filelist open for writing randomly and simultaneously? Then your request can't be fulfilled without implementing locking mechanisms, esp. in that other program. What you can do is keep your successes in an extra file, deduct / exclude those from the input file, and use the result for moving.
Dear RudiC,
When I run your #post3 script.
files are moved but processedfile.txt still contains the lines, which i need to update after move.
Once succesfully moved, these lines should not be present in the processedfile.txt filelist.
Regarding the [[-n $line ]] you seem keep to have, what are you trying to achieve?
Are you worried about blank input lines perhaps? If so, maybe this would be better:-
while read line
do
[[ -z "$line" ]] && continue # Jump round the loop for next item if we have nothing
mv $line $targetdir || echo "$line >&3" # rest of processing here
done < input-file 3> failure-list # Read input. Collect failure messages to a file
An alternate might be to strip them from your input file as you read it in something like:-
while read line
do
mv $line $targetdir || echo "$line" >&3
done < <(grep -Ev "^$" input-file) 3> failure-list # Feed in every non-blank line. Pattern excluded is <start-of-line><end-of-line>
This might depend on your OS and shell versions.
From RudiC's code (and above) you have a new file listing those that failed to move so that is what you would want. Hopefully it will be empty, unless you have duplicate files listed. If this is the case, you should sort the input file with the -u flag.
Editing the input file whilst still being read it is not only IO intensive for a large input file, but could have unpredictable results.
Robin
@rbatte1 #post15 for my knowledge could you please expalain (grep -Ev "^$" input-file part of the code.
---------- Post updated at 05:58 AM ---------- Previous update was at 05:56 AM ----------
RudiC, please help me to understand sed 's/\o047\| ->.*$//g; s/^/^/' | grep -vxf- file part of the code.
How do i update my processedfile.list file after successful move command.
So if my list file contains 1000 lines, and if all the lines read and successfully moved. then my list file should be empty. else have only the for the failed lines.
As I tried to point out, and I don't know how to phrase it more explicitly, the output of the script IS THE NEW LISTFILE, containing ALL THE FAILURES (i.e. having deleted the successful moves) and so REPLACING the OLD listfile, GIVEN your other process has not modified the old listfile in the meantime.
See the difference between posts #6 and#11 - from 10 entries in your listfile, 9 worked and have disappeared, one (...003...) failed and its entry is printed. You have to remove the OLD listfile - hopefully unmodified by the other process in the meantime - and put the printout in its place.
The sed mangles the mv -v output for the grep -v to follow which then removes the successful entries. cut does the same, btw.
Have 2 files contain:
File1 have lines:
/fstest/INVESTIG/Sadiq/TEST/file001.bin
/fstest/INVESTIG/Sadiq/TEST/file002.bin
/fstest/INVESTIG/Sadiq/TEST/file003.bin
/fstest/INVESTIG/Sadiq/TEST/file004.bin
File2 have lines:
/fstest/INVESTIG/Sadiq/TEST/file001.bin
/fstest/INVESTIG/Sadiq/TEST/file002.bin
/fstest/INVESTIG/Sadiq/TEST/file003.bin
/fstest/INVESTIG/Sadiq/TEST/file004.bin
/fstest/INVESTIG/Sadiq/TEST/file005.bin
/fstest/INVESTIG/Sadiq/TEST/file006.bin
/fstest/INVESTIG/Sadiq/TEST/file007.bin
/fstest/INVESTIG/Sadiq/TEST/file008.bin
/fstest/INVESTIG/Sadiq/TEST/file009.bin
/fstest/INVESTIG/Sadiq/TEST/file010.bin
I wan to validate the file and delete the common line from file2, so the file2 should contain only non matching lines.
How can i validate and achieve it ?
while read -r line;
do
if ! grep -Fxq File1 File2
then
"delete comman for the line in File2"
fi
done < /File1
man comm
how do i delete matching lines of one file from other in loop ?
---------- Post updated at 04:03 AM ---------- Previous update was at 03:58 AM ----------
comm -12 <(sort File1) <(sort File2)
Output:
/fstest/INVESTIG/Sadiq/TEST/file001.bin
/fstest/INVESTIG/Sadiq/TEST/file002.bin
/fstest/INVESTIG/Sadiq/TEST/file003.bin
/fstest/INVESTIG/Sadiq/TEST/file004.bin
But how do i delete it from File2 one by one ?