Hi to all,
I have thousand of files in a folder with names with format "FILE-YYYY-MM-DD-HHMM" for what I want to send the following AWK command
awk '/Code.*/' FILE-2014*
I'd like to separate all files that have the same date to a folder named with the corresponding date. For example, if I have these files
FILE-2014-10-30-1750
FILE-2014-10-30-2130
FILE-2014-10-31-2330
FILE-2014-11-02-0520
FILE-2014-11-02-1500
FILE-2014-11-02-1815
FILE-2014-11-12-1345
- I want to send "FILE-2014-10-30-1750" and "FILE-2014-10-30-2130" to folder "FILES-2014-10-30"
- I want to send "FILE-2014-10-31-2330" to folder "FILES-2014-10-31"
- I want to send "FILE-2014-11-02-0520", "FILE-2014-11-02-1500" and "FILE-2014-11-02-1815" to folder "FILES-2014-11-02"
- I want to send "FILE-2014-10-31-2330" to folder "FILES-2014-10-31"
Once the files are stored in their respective folder I want to run the AWK command above and generate an output file for each date, for example:
- The matched lines after run awk command for files of date "2014-10-30" store them in file "Codes-2014-10-30.txt"
- The matched lines after run awk command for files of date "2014-10-31" store them in file "Codes-2014-10-31.txt" etc, etc.
May somebody help me to achieve this please.
Thanks in advance.
What OS (including version) are you using? (If you don't know, show us the output from the command: uname -a ).
What have you tried to solve this problem?
Are the target directories you mentioned to be created in the directory that contains these files or in a different directory?
Are the output files from running the awk commands to be placed in the directory that originally contained the files, in the directory where the files being processed by each awk command have been moved, or in some other directory?
Do the directories to which the files are to be moved already exist? If so, are other files (that are not to be processed by the awk command for the files to be moved to that directory) in those directories?
What is the maximum number of files that could be moved into one of these target directories? (Or, more importantly, will invoking awk with the awk script and the absolute pathname of all of the moved files run into ARG_MAX limits? If there are enough files that that could be an issue, will the output from the commands:
cat FILE-2014-10-30-*| awk 'your awk script' > Codes-2014-10-30.txt
and:
awk 'your awk script' FILE-2014-10-30-* > Codes-2014-10-30.txt
be different?)
Hello Don,
I'm using Cygwin.
$ uname -a
CYGWIN_NT-6.1 1.7.28(0.271/5/3) 2014-02-09 21:06 i686 Cygwin
I know the basics of get basename of a file and I could get the name of all files in directory with a for like this:
for file in *; do; echo $(basename $file); done
but after this, I don't know how to get the date of each file and store all files of one date to a folder of the corresponding date.
After that, I don't have idea how to apply the awk command to the files inside each created directory and create an output file with the name of the respective date in dynamically way.
The target directories could go to /Processed folder that exists already and the output files generated by AWK command could go in /Processed too.
The directories to which the files are to be moved don't exist, since the name of each directory will be taken dynamically from the date of the files. For example, all files that are of date 2014-10-31, should go to folder "Codes-2014-10-31".
For each day the average of files to be moved is around 250 and in total the number of files is around 20,000.
And the number of days to be processed are 60 (2 months).
The output of these two commands, it seems to be exactly the same after trying with a file.
cat FILE-2014-10-30-*| awk 'your awk script' > Codes-2014-10-30.txt
and
awk 'your awk script' FILE-2014-10-30-* > Codes-2014-10-30.txt
Thanks again for any help.
Note that the command you said you were using to get part of your filenames:
for file in *; do; echo $(basename $file); done
is a very slow way of listing the files in your directory and should produce exactly the same output as the much faster:
ls -1
(the option is the digit one; not the lowercase letter ell).
You could use something like the following as a base for your script:
#!/bin/bash
destdir="/Processed"
#destdir="Processed"
here="$PWD"
lastdir=
runner() {
if [ "$lastdir" != "" ]
then cd "$lastdir"
# printf 'Running awk in directory "%s" on files:\n' "$lastdir"
# printf '\t"%s"\n' FILE*
awk 'your awk code here' FILE* > "$outfile"
# awk 'FNR == 1{print FILENAME}' FILE* > "$outfile"
# printf 'awk produced "%s" containing:\n' Codes*;cat Codes*;echo
cd "$here"
fi
lastdir="$dd"
outfile=Codes-"$base".txt
}
for f in FILE-[0-3][0-9][0-9][0-9]-[01][0-9]-[0-3][0-9]-[0-9][0-9][0-9][0-9]
do base=${f%-*}
base=${base#*-}
dd="$destdir/FILES-$base"
if [ "$dd" != "$lastdir" ]
then runner
fi
if [ ! -d "$dd" ]
then mkdir "$dd"
fi
mv "$f" "$dd"
done
runner
Testing for success of all of the awk , cd , mkdir , and mv commands snd taking appropriate actions if any of them fail is left as an exercise for the reader.
I left in the comments I used for testing in case you want it remove the octothorps to see status messages as it runs.
The version shown here assumes that your awk commands won't exceed ARG_MAX limits. If your awk commands do fail with "too many arguments" type failures, change the line:
awk 'your awk code here' FILE* > "$outfile"
to something like:
for p in FILE*
do cat "$p"
done | awk 'your awk code here' > "$outfile"
1 Like
Hello Don,
Thanks so much, you are a master!
I used destdir="Processed" instead of destdir="/Processed" and worked great testing with more or less 1000 files!!!.
I understand in a big percentage your code, but I have some doubts.
What does this expression "if [ ! -d "$dd" ]" mean/evaluate?
Regards
Indeed he is! 
Let me quote a few lines from man test :
NAME
test - check file types and compare values
SYNOPSIS
test EXPRESSION
test
[ EXPRESSION ]
[ ]
[ OPTION
...
! EXPRESSION
EXPRESSION is false
...
-d FILE
FILE exists and is a directory
In other words, if the destination directory specified in the variable $dd does not exist (then create it.)
1 Like
I'm glad my script is helping you.
And, as junior-helper said, the commands:
if [ ! -d "$dd" ]
then mkdir "$dd"
fi
checks to see of the destination directory ( $dd ) exists. If it does not exist, the then clause creates the directory. Since this test is performed each time through the loop, we expect that the directory will not exist the first time through the loop after the destination directory changes. On subsequent times through the loop with other files destined for the same target directory, it will already exist and no attempt will be made to create it again.
1 Like
Excellent junior-helper, excellent Don Cragun.
Many thanks for share in detail your knowledge and help others you don't even know which is even more laudable.
Best regards