How to define two digits variable in shell script?

emily · December 3, 2015, 8:45am

Dear Shell script Experts,
I am working on shell script which is defined here, qsub_seq.csh [1]. The purpose of this script is to read few input files (with defined starting index and last index) and make processing faster over server.
For some task, I had 1064 of input files, so I wrote another script, submitjob.sh [2] to make it quicker.
However, instantly I realized that it failed to work as the qsub_seq.csh script read the first digit of the input file and delete them all, though my input files have range from 0 till 1064, as given some examples of input files [3].

Any help, how to make qsub_seq.csh read two digits variable?

thank you in advance,
emily

[1] qsub_seq.csh
#!/bin/csh                                                                                                                                                                                                      
# $1 - code name                                                                                                                                                                                                
# $2 - analysis macro                                                                                                                                                                                           
# $3 - base directory                                                                                                                                                                                           
                                                                                                                                                            
# $4 - sample name                                                                                                                                                                                              
# $5 - starting index                                                                                                                                                                                           
# $6 - last index                                                                                                                                                                                               

rm -rf $4_files
mkdir $4_files
cp $2 $4_files/
cp qsub.sh $4_files/
cp temp $4_files/
cd $4_files
set n = $5
while ( $n <= $6 )
    ls $3/$4/$4_${n}*.root > $4_$2_$n
    ./qsub.sh $1 $2 $4_$2_$n
    @ n++
end
cd ../

[2] submitjobs.sh
#!/bin/sh                                                                                                                                                                                                       

Start=0
Step=9
Entries=1164

while [ $Start -lt $Entries ]
do
   if [ $Start -ge $Entries ]
   then
      break
   fi

   End=`expr $Start + $Step - 1`

   ./qsub_seq.csh AnalysisMacro analysisMacro.conf /nfs/dust/ DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8 $Start $End
   echo 'job submitted for ' $Start, $End
   Start=`expr $Start + $Step`
done

[3] 
--------------
/nfs/dust/emily/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8_0.root
/nfs/dust/emily/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8_151.root
/nfs/dust/emily/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8_299.root
/nfs/dust/emily/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8_2.root
/nfs/dust/emily/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8_99.root
/nfs/dust/emily/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8_9.root

agent.kgb · December 3, 2015, 8:57am

sorry for offtopic, but - Top Ten Reasons not to use C Shell

sea · December 3, 2015, 9:23am

From top to down:

submitjobs.sh
1) This part is not required:

if [ $Start -ge $Entries ]
   then
      break
   fi

As long it is lesser (-lt) it cannot exceed equal or greater than STOP, and will therefor break anyway.
2) Why not make STEP = 8, rather than END = START + STEP (9) - 1
3) I'm not familiar (enough) with cshell, but what happens in: ./qsub.sh $1 $2 $4_$2_$n
4) (one could write functions instead of different files)
5) You do overwrite the START variable before the loop ends, is that intended?

Last but not least, try to execute your scripts verbose / in debug mode, to see where the issues are, and how they look like:

 sh -x submitjobs.sh

And within ./qsub_seq.csh, change the shebang to #!/bin/csh -x (if that option is available for csh?)

hth

RudiC · December 4, 2015, 5:22am

Actually, I can't see how and where

. Possibly when passing parameters?
Wouldn't it make sense to collect the entire task into one single ( sh -, bash -, ksh -) script?

MadeInGermany · December 4, 2015, 11:57am

I see a problematic

  ls $3/$4/$4_${n}*.root > $4_$2_$nby

Perhaps you can simply omit the * ?

emily · December 15, 2015, 3:42am

Hello all,
Thanks for your replies, it helped me to dig it further.
Now, I realize I shall be able to fix it, if I could figure out how to provide the command line in a shell script to create directory/anotherDirectory. Any hints are appreciated..
Basically, I want to do somethiing like this :

#!/bin/sh                                                                                                                                                                                                       

Start=0
Step=9
Entries=1164

./qsub_seq.csh  file.conf /inifitla/input/file/location Directory/subDir$Start 0 2

Basically, I want to redirect the output to '

Directory/subDir$Start

' which I have to create.

Greetings,
emily

RudiC · December 15, 2015, 4:45am

man mkdir ?

emily · December 15, 2015, 4:52am

Hello,
Yes, I tried this as following:

#!/bin/sh

Start=0
Step=9
Entries=1164

mkdir DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/
mkdir DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir$Start
./qsub_seq.csh AnalysisMacro analysisMacro_tmvaEMu.conf /inputfile/location DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir$Start 0 2

However, It failed to work. The output was as following:

DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0
DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_0: No such file or directory.
./qsub.sh: line 5: DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_0.zsh: No such file or directory
rm: cannot remove `DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_0.out': No such file or directory
chmod: cannot access `DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_0.zsh': No such file or directory
Unable to read script file because of error: error opening DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_0.zsh: No such file or directory
DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_1: No such file or directory.
./qsub.sh: line 5: DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_1.zsh: No such file or directory
rm: cannot remove `DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_1.out': No such file or directory
chmod: cannot access `DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_1.zsh': No such file or directory
Unable to read script file because of error: error opening DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_1.zsh: No such file or directory
DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_2: No such file or directory.
./qsub.sh: line 5: DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_2.zsh: No such file or directory
rm: cannot remove `DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_2.out': No such file or directory
chmod: cannot access `DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_2.zsh': No such file or directory
Unable to read script file because of error: error opening DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_2.zsh: No such file or directory

Probably, I need to look for better solution of executing this script.

RudiC · December 15, 2015, 5:28am

No surprise you're getting an error msg - you're constructing the files' paths incorrectly:

DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/Subdir0_analysisMacro_tmvaEMu.conf_0: No such file or directory.

At least, there's a / missing after "Subdir0".