Bash to create sub directories from specific file extension

Shell Programming and Scripting
1

In the bash below I am trying to create sub-directories inside a directory from files with specific .bam extensions. There may be more then one $RDIR ing the directory and the .bam file(s) are trimmed (removing the extension and IonCode_0000_) and the result is the folder name that is saved in $RDIR or one level up from where the original files are found. There may be multiple .bam files but they are always the same format. I made comments as well and ran set -x and think the find is not working as expected but am not able to fix it . Thank you :).

structure of /home/cmccabe/Desktop/folder --- this is $DIR ---

R_2019_00_00_00_00_00_xxxx_xx-0000-00  --- this is $RDIR ---
       BAM   ---subdirectory---
       IonCode_0241_19-0000-Last-First.bam.bai
       IonCode_0241_19-0000-Last-First.bam
       IonCode_0243_19-0001-Las-Fir.bam.bai
       IonCode_0243_19-0001-Las-Fir.bam
     QC    ---subdirectory---

after script structure of /home/cmccabe/Desktop/folder --- this is $DIR ---

R_2019_00_00_00_00_00_xxxx_xx-0000-00  --- this is $RDIR ---
     BAM                  ---subdirectory---
     19-0000-Last-First   ---subdirectory---
     19-0001-Las-Fir      ---subdirectory---
     QC                   ---subdirectory---

set -x

set -x
cmccabe@DTV-A5211QLM:~$ DIR=/home/cmccabe/Desktop/sub   ## define data directory path
+ DIR=/home/cmccabe/Desktop/sub
cmccabe@DTV-A5211QLM:~$ for RDIR in "$DIR"/R_2019* ; do  ## start processing matching "R_2019*" to operate on desired directory and expand
> echo "$RDIR"
>  cd "$RDIR"/BAM   ## change directory to subfolder inside $RDIR
>   bam=$(find . -type f -name "*.bam")   # extract .bam
>   sample="$(echo $bam|cut -d_ -f3-)" # remove before second underscore
>   mkdir -p "${sample%.*}" && mv "$sample" "RDIR"/"${sample%.*}"  ## make directory of sample id one level up
> cd ..
> done  ## close loop
+ for RDIR in '"$DIR"/R_2019*'
+ echo /home/cmccabe/Desktop/sub/R_2019_01_23_13_22_58_user_S5-0271-93
/home/cmccabe/Desktop/sub/R_2019_01_23_13_22_58_user_S5-0271-93
+ cd /home/cmccabe/Desktop/sub/R_2019_01_23_13_22_58_user_S5-0271-93/BAM
++ find . -type f -name '*.bam'
+ bam='./IonCode_0251_19-0003-La-Fi.bam
./IonCode_0243_19-0001-Las-Fir.bam
./IonCode_0255_190319-Control.bam
./IonCode_0241_19-0000-Last-First.bam'
++ echo ./IonCode_0251_19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam
++ cut -d_ -f3-
+ sample='19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam'
+ mkdir -p '19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First'
+ mv '19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam' 'RDIR/19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First'
mv: cannot stat �19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam': No such file or directory
+ cd ..
+ for RDIR in '"$DIR"/R_2019*'
+ echo /home/cmccabe/Desktop/sub/R_2019_01_23_13_22_58_user_S5-0271-96
/home/cmccabe/Desktop/sub/R_2019_01_23_13_22_58_user_S5-0271-96
+ cd /home/cmccabe/Desktop/sub/R_2019_01_23_13_22_58_user_S5-0271-96/BAM
++ find . -type f -name '*.bam'
+ bam='./IonCode_0251_19-0003-La-Fi.bam
./IonCode_0243_19-0001-Las-Fir.bam
./IonCode_0255_190319-Control.bam
./IonCode_0241_19-0000-Last-First.bam'
++ echo ./IonCode_0251_19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam
++ cut -d_ -f3-
+ sample='19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam'
+ mkdir -p '19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First'
+ mv '19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam' 'RDIR/19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First'
mv: cannot stat �19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam': No such file or directory
+ cd ..
2

I modified the find and added a loop to mkdir and got the expected results. Just wanted to share in case it helps anyone else in the future. Thank you :).

DIR=/home/cmccabe/Desktop/sub   ## define data directory path
for RDIR in "$DIR"/R_2019* ; do  ## start processing matching "R_2019*" to operate on desired directory and expand
 cd "$RDIR"/BAM   ## change directory to subfolder inside $RDIR
i=$(find . -type f -name "*.bam" -print | while read f;do echo "$f" | cut -d_ -f2-;done| cut -f 1 -d '.')  # extract .bam
for x in $i
        do mkdir -p $RDIR/$x
        done
done  ## close loop
1 Like
3 
$(find * -type f -name "*.bam" | cut -d_ -f2 | cut -d. -f1)

--- Post updated at 18:32 ---

endless running through directories means only one thing, you don't know where you will when will creating catalogs

find "$RDIR"/BAM -type f -name "*.bam" | cut -d_ -f2 | cut -d. -f1 |  xargs -I {} mkdir -p "$RDIR/"{}

--- Post updated at 18:44 ---

In any case, it is better not to run on items twice.

do
    cd "$RDIR"/BAM
    find * -type f -name "*.bam" | cut -d_ -f2 | cut -d. -f1 |  xargs -I {} mkdir -p "$RDIR/"{}
done