Bash and Awk for creating directories and moving files

I have a security system that FTPs the camera files to my machine, however I want to sort the pictures (taken every 30s) into directories by hour.

Every picture uses the following file format.
yymmddhhmmsstt.jpg (where tt is the milliseconds)

I am thinking the for loop is best

for file in *.jpg
do
?here?
done

I know its very vague, however the end result is I am guessing use awk (or better?) to create variables from the yy mm dd hh parts of the filenames, then creating directories based on those variables nested with a check to see if the directory exists, creating the directory if it doesnt.

i.e. (the way I imagine it working, I dont know how to actually construct the script to do this)

for file in 08072400123200.jpg
do
%y=08
%m=07
%d=24
%h=00
if not exist directory %y - mkdir %y
if not exist directory %y/%m - mkdir %y/%m
if not exist directory %y/%m/%d - mkdir %y/%m/%d
if not exist directory %y/%m/%d/%h - mkdir %y/%m/%d/%h
mv %y%m%d%h*.jpg %y/%m/%d/%h
done

Moving all the pictures taken within that hour into the respective subdirectories.

Any help would be much appreciated.

using bash, you should start reading up here. It shows you how to get substrings.

What I have so far, that I imagine should work (but doesnt) is (and isnt very elegant or sane)

#!/bin/bash

for file in *.jpg; do

set yy = `echo $file | awk '{split($0,a,""); print a[1]a[2]}'`
set mm = `echo $file | awk '{split($0,a,""); print a[3]a[4]}'`
set dd = `echo $file | awk '{split($0,a,""); print a[5]a[6]}'`
set hh = `echo $file | awk '{split($0,a,""); print a[7]a[8]}'`

if [ -d $yy ]; then
        if [ -d $yy/$mm ]; then
                if [ -d $yy/$mm/$dd ]; then
                        if [ -d $yy/$mm/$dd/$hh ]; then
                                mv $yy$mm$dd$hh*.jpg $yy/$mm/$dd/$hh/$file
                        elif
                                mkdir $yy/$mm/$dd/$hh
                                exit 1
                        fi
                elif
                        mkdir $yy/$mm/$dd
                        exit 1
                fi
        elif
                mkdir $yy/$mm
                exit 1
        fi
elif
        mkdir $yy
        exit 1
fi

done

Even modifying for various shell stupidity doesnt help

./filter.sh: line 17: syntax error near unexpected token `fi'
./filter.sh: line 17: ` fi'

#!/bin/bash -x

for file in *.jpg; do

YY = "echo $file | awk '{split($0,a,""); print a[1]a[2]}'"
MM = "echo $file | awk '{split($0,a,""); print a[3]a[4]}'"
DD = "echo $file | awk '{split($0,a,""); print a[5]a[6]}'"
HH = "echo $file | awk '{split($0,a,""); print a[7]a[8]}'"

if [ -d $YY ]; then
        if [ -d $YY/$MM ]; then
                if [ -d $YY/$MM/$DD ]; then
                        if [ -d $YY/$MM/$DD/$HH ]; then
                                mv $YY$MM$DD$HH*.jpg $YY/$MM/$DD/$HH/$file
                        elif
                                mkdir $YY/$MM/$DD/$HH
                        fi
                elif
                        mkdir $YY/$MM/$DD
                fi
        elif
                mkdir $YY/$MM
        fi
elif
        mkdir $YY
fi

done

We do not have to test for individual directory level 'cos if the bottom directory exists, the parent level should exist. Also, mkdir -p will make all the non-existing parent directories. Also, I introduce 'short circurt' && to ensure directory exist before I move the file.

We can also avoid all the repeating code using in extracting yy/mm/dd/hh by using 'set --' and sed. sed will change 2 digits with 2 digits + space so that it can put the result back to "set --" to set the positional variables accordingly

This is my contribution, it should work (even on sh)

for i in *.jpg
do
   # yy is $1, mm is $2, dd is $3, hh is $4
   set -- `echo $i | sed -e 's/\([0-9][0-9]\)/\1 /g'`
   dir="$1/$2/$3/$4"
   [ ! -d $dir ] && mkdir -p $dir && mv $i $dir
done

Well, its working (for those out there that stumble across this site for a similar script)

Its not elegant, sane or other ... but it works.

If the better experienced here can clean it up and solve the elegant/sane issues then it would be great :slight_smile:

#!/bin/bash -x

for file in *.jpg; do

YY=`echo $file | awk '{split($0,a,""); print a[1]a[2]}'`
MM=`echo $file | awk '{split($0,a,""); print a[3]a[4]}'`
DD=`echo $file | awk '{split($0,a,""); print a[5]a[6]}'`
HH=`echo $file | awk '{split($0,a,""); print a[7]a[8]}'`

if [ -d $YY ]
then
        if [ -d $YY/$MM ]
        then
                if [ -d $YY/$MM/$DD ]
                then
                        if [ -d $YY/$MM/$DD/$HH ]
                        then
                                mv $YY$MM$DD$HH*.jpg $YY/$MM/$DD/$HH
                        else
                                mkdir $YY/$MM/$DD/$HH
                        fi
                else
                        mkdir $YY/$MM/$DD
                fi
        else
                mkdir $YY/$MM
        fi
else
        mkdir $YY
fi

done

Shorther:

for i in *.jpg;do d=${i:0:8};test -d $d || mkdir $d ;mv $i $d;done

Your script is absolutely perfect, I love it ... except ...

the last --> [ ! -d $dir ] && mkdir -p $dir && mv $i $dir <-- wouldnt actually move the file if the directory existed already, so I added an extra line below with just --> mv $i $dir <-- in place and that solved that part, so it now becomes, although I would imagine this will now generate an error if the directory it wants to create doesnt exist (but shouldnt error for subsequent files - I think)

#!/bin/bash -x

for i in *.jpg
do
   # yy is $1, mm is $2, dd is $3, hh is $4
   set -- `echo $i | sed -e 's/\([0-9][0-9]\)/\1 /g'`
   dir="$1/$2/$3/$4"
   [ ! -d $dir ] && mkdir -p $dir && mv $i $dir
   mv $i $dir
done

Awesome single line version, perfect except the lack of nested directories.

Instead of a single directory 08072511 I need them nested like

>08
->07
-->25
--->11

I was trying to be 'smart', too many 'short circuit' make me short circuit too.
it should look like this:

for i in *.jpg
do
   # yy is $1, mm is $2, dd is $3, hh is $4
   set -- `echo $i | sed -e 's/\([0-9][0-9]\)/\1 /g'`
   dir="$1/$2/$3/$4"
   [ ! -d $dir ] && mkdir -p $dir
   mv $i $dir
done

You may even want to ensure *.jpg to be your patten by:

for i in [0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9].jpg
do
...

Not sure whether bash has a more elegant way to pick up your files in the for loop, but the above [0-9]... should definitely work for sh.

As for danmero contribution, I couldn't get it working in Solaris bash

$ i=090807060504.jpg
$ d=${i:0:8}
bad substitution
$ echo $SHELL
/bin/bash
$ uname -a
SunOS chihung 5.10 Generic_118833-36 sun4u sparc SUNW,UltraSPARC-IIi-cEngine

In cygwin, although it does not throw exception, the variable d is equivalent to extracting the first 8 digits. We still need to turn that into directory path before we can make the hierarchical tree structure

$ i=090807060504.jpg
$ d=${i:0:8}
$ echo $d
09080706
$ uname -a
CYGWIN_NT-5.1 chihung 1.5.25(0.156/4/2) 2007-12-14 19:21 i686 Cygwin

Danmero, did I miss anything

Ops, fast reading :wink:

for i in *.jpg;do d=.$(sed 's/\(..\)/\/\1/g' <<< ${i:0:8});test -d $d || mkdir -p $d ;mv $i $d;done

@chihung

$ uname -a
Linux test 2.6.18-6-686 #1 SMP Sun Feb 10 22:11:31 UTC 2008 i686 GNU/Linux

Try...

for i in *.jpg; do d=$(echo $i|cut -c-8|fold -2|paste -s -d /); mkdir -p $d; mv $i $d; done