Variable not passed to the sed command

Hello,
I am writing a script [1] which is not giving the desired result. When I check the content of the 'InputFile_009_0.sh'[2], it shows following with missing Index in this command

sed -i "s/L1ITMBLT.root/L1ITMBLT_"".root/g" run_DttfFromCombinedPrimitives_cfg.py

of [2].

Any help?

[1]

#!/bin/sh                                                                                                                    
CONFIGFILE=run_DttfFromCombinedPrimitives_cfg.py

for FILE in *txt
do
    echo $FILE
    ScriptName=$(echo $FILE | awk '{split($FILE, a, ".txt"); print a[1]}')
    cat > $ScriptName.sh <<EOF                                                                                               
#!/bin/sh                                                                                                                    
sed -i "s/InputFile[a-zA-Z0-9]*/${FILE}/g" $CONFIGFILE                                                                       
Index=$(echo $FILE | awk '{match($0,/\_.*\_/);print substr($0,RSTART,RLENGTH+1)}')   
echo $Index                                        
sed -i "s/L1ITMBLT.root/L1ITMBLT_"${Index}".root/g" $CONFIGFILE                                                              
#cmsRun $CONFIGFILE                                                                                                          
EOF                                                                                                                          
done

-------------------------------------------------------------------------
where FILE in the directory is:
InputFile_009_0.txt
InputFile_009_1.txt
InputFile_009_2.txt
InputFile_009_3.txt
InputFile_009_4.txt

[2]

#!/bin/sh                                                                                                                    
sed -i "s/InputFile[a-zA-Z0-9]*/InputFile_009_0.txt/g" run_DttfFromCombinedPrimitives_cfg.py
Index=_009_0
sed -i "s/L1ITMBLT.root/L1ITMBLT_"".root/g" run_DttfFromCombinedPrimitives_cfg.py
#cmsRun run_DttfFromCombinedPrimitives_cfg.py

The variable Index needs to be set outside the here-document, if you want to use it inside the here-document.

Hello emliy,

Not tested though but could you please try following command and let me know if this helps.

sed -i 's/L1ITMBLT.root/L1ITMBLT_${Index}.root/g' $CONFIGFILE

Thanks,
R. Singh

Did not work...

---------- Post updated at 09:20 AM ---------- Previous update was at 09:16 AM ----------

I modified the script as following, did not help either

#!/bin/sh                                                                                                                    
CONFIGFILE=run_DttfFromCombinedPrimitives_cfg.py
Index=trash

for FILE in *txt
do
    echo $FILE
    ScriptName=$(echo $FILE | awk '{split($FILE, a, ".txt"); print a[1]}')
    cat > $ScriptName.sh <<EOF                                                                                               
#!/bin/sh                                                                                                                    
sed -i "s/InputFile[a-zA-Z0-9]*/${FILE}/g" $CONFIGFILE                                                                       
Index=$(echo $FILE | awk '{match($0,/\_.*\_/);print substr($0,RSTART,RLENGTH+1)}')                                           
sed -i 's/L1ITMBLT.root/L1ITMBLT_$Index.root/g' $CONFIGFILE                                                                  
#cmsRun $CONFIGFILE                                                                                                          
EOF                                                                                                                                                                                                                               

done

the output is following

#!/bin/sh                                                                                                                    
sed -i "s/InputFile[a-zA-Z0-9]*/InputFile_009_0.txt/g" run_DttfFromCombinedPrimitives_cfg.py
Index=_009_0
sed -i 's/L1ITMBLT.root/L1ITMBLT_trash.root/g' run_DttfFromCombinedPrimitives_cfg.py
#cmsRun run_DttfFromCombinedPrimitives_cfg.py 

Hello emily,

Could you please try following command this should work.

sed -i s/L1ITMBLT.root/L1ITMBLT_$Index.root/g $CONFIGFILE

No need of using ' in it.

Thanks,
R. Singh

Does not work either, output is following:

sed -i s/L1ITMBLT.root/L1ITMBLT_.root/g run_DttfFromCombinedPrimitives_cfg.py

Well then it DOES work, doesn't it ( L1ITMBLT_trash.root )?

i want the output as following:

sed -i 's/L1ITMBLT.root/L1ITMBLT_009_0.root/g' run_DttfFromCombinedPrimitives_cfg.py

which is the value of Index taken from the awk command

Yes but you will need to assign the value to the Index variable outside the here document. You cannot assign a value to a variable inside...

--
An alternative is to escape the $ sign inside the here document and let the generated script substitute the variable:

sed -i 's/L1ITMBLT.root/L1ITMBLT_\$Index.root/g' $CONFIGFILE

then your generated file would become:

#!/bin/sh                                                                                                                    
sed -i "s/InputFile[a-zA-Z0-9]*/InputFile_009_0.txt/g" run_DttfFromCombinedPrimitives_cfg.py
Index=_009_0
sed -i 's/L1ITMBLT.root/L1ITMBLT_$Index.root/g' run_DttfFromCombinedPrimitives_cfg.py

Yes, it fixed the problem..
Would like to know why the new value could not be assigned to the Index inside ?

thanks again,
emily

I had edited the post in the mean time, have a look..

My code setup seem to be reaching it completeness to perform the desired task..thanks a lot everyone for great help
I always find useful and timely help here..

Dear Scrutinizer,
I just have a small question, which is "why the new value could not be assigned to the Index inside ?"

thank you,
Emily

You're welcome...

From man bash:

So everything inside the here document is standard input to the cat command (not commands that are executed)

Since EOF is unquoted:

I suppose the problem comes from a misunderstanding about what is done when in the shells evaluation process.

When you write a process substitution in a here-document:

while read line ; do
     echo $line
done <<EOF
$(some_command)
EOF

The following happens: a subshell is loaded and "some_command" is executed in this subshell. The output (to stdout) of this process is taken and the process substitution is replaced with this outcome. Lets say the outcome is "blahblah" then the shell would arrive at this:

while read line ; do
     echo $line
done <<EOF
blahblah
EOF

Only now this resulting here document is fed to the (compound)-command. I used a while-loop for the example but it could be any other single or complex command too. Notice, that "blahblah" is not treated as a command - it is treated as a string! This will perfectly work even though the shell wouldn't know what to make of a command "blahblah".

Now consider this:

while read line ; do
     echo $line
done <<EOF
x=blahblah
EOF

Yes, it looks like variable "x" would be assigned some value, but in fact this is not treated as a command either - it is just a string, the same way "blahblah" alone was just a string!

You had this string in the here-document and expected it to be executed - but this is not the case. The shell only evaluates the here-document, which means that process substitution gets done and variables are expanded:

var="blahblah"
while read line ; do
     echo $line
done <<EOF
x=$var
EOF

In this example the string "$var" willl be replaced by the content of the variable "var", which is "blahblah", therefore the resulting string passed to the while-loop will again be "x=blahblah", but this will still not be executed and a variable "x" will still not have the value "blahblah" at all.

What you can do to achieve what you obviously want is the following:

[...your script....] <<EOF
[...here-document....]
sed -i 's/L1ITMBLT.root/L1ITMBLT_$(echo $FILE | awk '{match($0,/\_.*\_/);print substr($0,RSTART,RLENGTH+1)}').root/g' $CONFIGFILE 
[...more here-document....]
EOF

but i admit this looks ugly and i suggest not to do it at all. My suggestion is based on ksh (don't know if you have that) and looks like this (only sketched out). Note that for-loops with an undefined number of elements is a bad idea in any shell because it can break if "*txt" evaluates to too many files. I used a while-loop therefore. I also removed the awk-scripts where they could be replaced by simple shell-expansion:

#! /bin/ksh
typeset CONFIGFILE=run_DttfFromCombinedPrimitives_cfg.py
typeset FILE=""
typeset Scriptname=""

ls *txt |\
while read FILE ; do
    echo $FILE
    ScriptName="${FILE%*\.txt}.sh"
    Index=$(echo $FILE | awk '{match($0,/\_.*\_/);print substr($0,RSTART,RLENGTH+1)}')

    exec 3>"$ScriptName"
    print -u3 - "#!/bin/sh"
    print -u3 - "sed -i 's/InputFile[a-zA-Z0-9]\*/${FILE}/g' $CONFIGFILE"
    print -u3 - "sed -i \"s/L1ITMBLT.root/L1ITMBLT_${Index}.root/g $CONFIGFILE\""
    print -u3 - "#cmsRun"
    print -u3 - $CONFIGFILE"
    exec 3>&-

done

I admit, i couldn't make out what the awk-orgy is supposed to do, so i let it in place. You might want to replace it with a shell expansion eventually.

I hope this helps.

bakunin