Script to run command one by one

misterx12345 · April 13, 2013, 7:50am

Hi,

I have run a lot of commands one after one,
but I can't run them simultaneous.
One command has to run and when finished second command has to run etc.
Also I would like to save all the outputs created by the commands in a file.
The commands can sometimes take hours, so it is also necessary to run them in the background or the whole script.
I am running bash shell.
The commands are very long, so I put them in these files: cmd1, cdm2, cmd3 etc
SO what I need is that the scripts runs and executes cmd1 and when finished , execute cmd2 etc.
This is what I got so far:

#! /bin/bash

# run commands in sequence

echo "Running cmd1"
sh cmd1 > output.txt 2>&1

wait

echo "Running cmd2"
sh cmd2 >> output.txt 2>&1


wait

echo "Running cmd3"
sh cmd3 >> output.txt 2>&1


echo " Scripts are finished !!"

Is it possible that the script itself fills in the number for the cmd files?
for example it will start with cmd1 and then cmd2 etc and stop until it can't find more?

Don_Cragun · April 13, 2013, 9:27am

Is this a homework assignment?

misterx12345 · April 13, 2013, 10:41am

This is for production environment.

Maybe this is better, but how would this run all the commands :

#! /bin/bash

for i in {1..30} ;

do

echo "##########Running cmd$i ##########$(date)"  > output.txt
sh cmd$i >> output.txt 2>&1

Don_Cragun · April 13, 2013, 11:16am

The script that you suggested will overwrite the output of all previously executed scripts with the output of the last script to run. The wait commands in your script don't do anything. (You didn't start the cmd* scripts asynchronously and it wouldn't make any difference if you did.) If you have some long running cmd* scripts, run this shell script in the background; not the jobs started by the script. You might want to try something like:

#!/bin/bash
i=1
while [ -f cmd$i ]
do      sh cmd$i > cmd$i_stdout.txt 2> cmd$i_stderr.txt
        i=$((i + 1))
done

Note that you specified that this is a bash script, but you're executing the cmd* scripts using sh (rather than bash). If the scripts to be run are executable scripts, you could omit the "sh" in red in the above script and let a #! line in each script specify the shell to be used for any script that doesn't want to use the system's default shell. This script will run any number of cmd* scripts as long as their names have consecutive integer values (starting with "1") at the ends of the names.

PS I didn't notice that you had used >> instead of > for later jobs, so you won't be overwriting the output. You may still want to consider the redirection methods shown above to separate stdout and stderr for each script into separate files. I also skipped the echoes you had showing the progression of scripts being executed. If you want them, I assume you know where to add them. (If not; let me know.)

misterx12345 · April 13, 2013, 12:17pm

Thx Don for your quick reply.
Yes, I am using the bash, but should I use something else to run the commands?
I did run your script, but it didn't do anything.
Didn't get an error and no output files.
Seems like it didn't do anything.

don cragun:

The script that you suggested will overwrite the output of all previously executed scripts with the output of the last script to run. The wait commands in your script don't do anything. (You didn't start the cmd* scripts asynchronously and it wouldn't make any difference if you did.) If you have some long running cmd* scripts, run this shell script in the background; not the jobs started by the script. You might want to try something like:
#!/bin/bash
i=1
while [ -f cmd$i ]
do      sh cmd$i > cmd$i_stdout.txt 2> cmd$i_stderr.txt
   i=$((i + 1))
done
Note that you specified that this is a bash script, but you're executing the cmd* scripts using sh (rather than bash). If the scripts to be run are executable scripts, you could omit the "sh" in red in the above script and let a #! line in each script specify the shell to be used for any script that doesn't want to use the system's default shell. This script will run any number of cmd* scripts as long as their names have consecutive integer values (starting with "1") at the ends of the names.

PS I didn't notice that you had used >> instead of > for later jobs, so you won't be overwriting the output. You may still want to consider the redirection methods shown above to separate stdout and stderr for each script into separate files. I also skipped the echoes you had showing the progression of scripts being executed. If you want them, I assume you know where to add them. (If not; let me know.)

---------- Post updated at 08:17 AM ---------- Previous update was at 07:33 AM ----------

Apologies, your command does work.
thx very much !!
but is it also possible to echo which command is going to run and date in front of every command in the log file?

don cragun:

The script that you suggested will overwrite the output of all previously executed scripts with the output of the last script to run. The wait commands in your script don't do anything. (You didn't start the cmd* scripts asynchronously and it wouldn't make any difference if you did.) If you have some long running cmd* scripts, run this shell script in the background; not the jobs started by the script. You might want to try something like:
#!/bin/bash
i=1
while [ -f cmd$i ]
do      sh cmd$i > cmd$i_stdout.txt 2> cmd$i_stderr.txt
   i=$((i + 1))
done
Note that you specified that this is a bash script, but you're executing the cmd* scripts using sh (rather than bash). If the scripts to be run are executable scripts, you could omit the "sh" in red in the above script and let a #! line in each script specify the shell to be used for any script that doesn't want to use the system's default shell. This script will run any number of cmd* scripts as long as their names have consecutive integer values (starting with "1") at the ends of the names.

PS I didn't notice that you had used >> instead of > for later jobs, so you won't be overwriting the output. You may still want to consider the redirection methods shown above to separate stdout and stderr for each script into separate files. I also skipped the echoes you had showing the progression of scripts being executed. If you want them, I assume you know where to add them. (If not; let me know.)

Don_Cragun · April 13, 2013, 12:59pm

I can't tell you what interpreter you should use to run your commands other than to say that the interpreter used has to be an interpreter that recognizes the commands it is being asked to run. You are the one who knows what is in the cmd* files. You are the one who knows if those files have a first line of the form #!interpreter . You are the one who knows if the cmd* files have been marked executable or are just readable. Without this information, I don't know what you should do. It just seemed strange that you were writing this script using bash, but using a Bourne shell to run the cmd* files. (Of course, you haven't said what system you're using; so sh and bash may be the same shell.)

Do you want all of the output from all of the cmd* files in one output file? Do you want the standard output from all of the cmd* files in one output file and standard error output from all of the cmd* files in a different output file? Or, do you want the standard output and standard error output for each cmd* file in separate output files (like the script I gave you currently does)?

Do you want the output showing which cmd* file is being started and the timestamp written to the same file as the standard output for the cmd* file that is about to run, to the standard output of the entire script, or to a different log file?

misterx12345 · April 13, 2013, 3:56pm

I am using bash.
The reason for using the sh was something I found on the internet, wasn't aware it was bourne shell.
I just want to direct all the output to one file, which I did manage by replacing " > cmd$i_stdout.txt 2> cmd$i_stderr.txt " with " > output.txt 2>&1"
I just need now need the timestamp when every cmd* commands runs in a different file (this is to see how long the commands run).

we get these commands usually in one file from a different department, they don't have enough privileges on their account to execute these command
every command creates and *.csv file, we just copy the commands one by one into a file (commands are very long) and then execute it and send the *.csv file by email and
in the beginning we just got couple of commands, but now we are getting a lot and
these commands can sometime take hours to run
this is too much effort, because sometimes one command can take hours to run and we have to wait for the command to finish, before we can move on to the next one,
I want to automate this as much as possible and the script you gave does the job. We are working on Solaris 10 and I am using the bash shell.
Thx.

don cragun:

I can't tell you what interpreter you should use to run your commands other than to say that the interpreter used has to be an interpreter that recognizes the commands it is being asked to run. You are the one who knows what is in the cmd* files. You are the one who knows if those files have a first line of the form #!interpreter . You are the one who knows if the cmd* files have been marked executable or are just readable. Without this information, I don't know what you should do. It just seemed strange that you were writing this script using bash, but using a Bourne shell to run the cmd* files. (Of course, you haven't said what system you're using; so sh and bash may be the same shell.)

Do you want all of the output from all of the cmd* files in one output file? Do you want the standard output from all of the cmd* files in one output file and standard error output from all of the cmd* files in a different output file? Or, do you want the standard output and standard error output for each cmd* file in separate output files (like the script I gave you currently does)?

Do you want the output showing which cmd* file is being started and the timestamp written to the same file as the standard output for the cmd* file that is about to run, to the standard output of the entire script, or to a different log file?

Don_Cragun · April 13, 2013, 4:54pm

misterx12345:

I am using bash.
The reason for using the sh was something I found on the internet, wasn't aware it was bourne shell.
I just want to direct all the output to one file, which I did manage by replacing " > cmd$i_stdout.txt 2> cmd$i_stderr.txt " with " > output.txt 2>&1"
I just need now need the timestamp when every cmd* commands runs in a different file (this is to see how long the commands run).

we get these commands usually in one file from a different department, they don't have enough privileges on their account to execute these command
every command creates and *.csv file, we just copy the commands one by one into a file (commands are very long) and then execute it and send the *.csv file by email and
in the beginning we just got couple of commands, but now we are getting a lot and
these commands can sometime take hours to run
this is too much effort, because sometimes one command can take hours to run and we have to wait for the command to finish, before we can move on to the next one,
I want to automate this as much as possible and the script you gave does the job. We are working on Solaris 10 and I am using the bash shell.
Thx.

It seems to me that putting the output from all of the cmd* scripts in a single file is just making your life harder (since you have to manually find the end of each command file's output and write it to a separate file to mail it back to the user), but I have done as you requested. I did sneak in two status lines in the output.txt file that should make it easier to separate the output after all of the cmd* files have finished and have also included the exit code in the closing separator line in output.txt since I would think your users will want that information.

This script logs when a script starts, when it ends (and includes its exit status), and when the last script ends to the file log.txt. Both log.txt and output.txt are cleared when you start this script; so don't run it a second time before you extract all of the data you need from output.txt. This seems dangerous to me, but it is what you requested.

#!/bin/bash
# Throw away output from previous runs.
> output.txt

# Redirect this script's output to log file.
exec > log.txt 2>&1

# Loop through all available cmd scripts.
i=1
while [ -f cmd$i ]
do      date "+cmd$i starting @ %c"
        date "+================ cmd$i start ================ %c" >> output.txt
        sh cmd$i >> output.txt 2>&1
        ec=$?
        date "+============= cmd$i exit code $ec ============= %c" >> output.txt
        date "+cmd$i exit code $ec @ %c" >> log.txt
        i=$((i + 1))
done
date "+Scripts finished @ %c"

misterx12345 · April 13, 2013, 5:25pm

Thank you very much sir !!!
I will try this when I get in the office on Thursday.

Have a nice weekend.

don cragun:

It seems to me that putting the output from all of the cmd* scripts in a single file is just making your life harder (since you have to manually find the end of each command file's output and write it to a separate file to mail it back to the user), but I have done as you requested. I did sneak in two status lines in the output.txt file that should make it easier to separate the output after all of the cmd* files have finished and have also included the exit code in the closing separator line in output.txt since I would think your users will want that information.

This script logs when a script starts, when it ends (and includes its exit status), and when the last script ends to the file log.txt. Both log.txt and output.txt are cleared when you start this script; so don't run it a second time before you extract all of the data you need from output.txt. This seems dangerous to me, but it is what you requested.
#!/bin/bash
# Throw away output from previous runs.
> output.txt
 
# Redirect this script's output to log file.
exec > log.txt 2>&1
 
# Loop through all available cmd scripts.
i=1
while [ -f cmd$i ]
do      date "+cmd$i starting @ %c"
   date "+================ cmd$i start ================ %c" >> output.txt
   sh cmd$i >> output.txt 2>&1
   ec=$?
   date "+============= cmd$i exit code $ec ============= %c" >> output.txt
   date "+cmd$i exit code $ec @ %c" >> log.txt
   i=$((i + 1))
done
date "+Scripts finished @ %c"

misterx12345 · April 18, 2013, 6:15am

The script works great.
I am just trying to figure out the script.

"exec > log.txt 2>&1"
what is the function of exec , shows all output of everything being executed?
"do date "+cmd$i starting @ %c" "
how does the script know how to send this to the log.txt?
what is "ec=$? "

i=$((i + 1))
this tells the scripts to restart the commands but with the next cmd file?

Thx.

Don_Cragun · April 18, 2013, 9:55am

The general format for the exec command:

    exec command [redirections]

overlays the current shell script with the command named by command and performs the input and/or output redirections specified by redirections for that command. When no command is specified, the exec command performs the specified redirections for the current script. So, in this case:

exec > log.txt 2>&1

no command is specified, so the standard output of the remaining commands in this script that do not otherwise redirect their standard output will be written to the file named log.txt and any data written to standard error by the remaining commands in this script that do not otherwise redirect their standard error output will be sent to the same file. Since the redirection to log.txt uses ">" (rather than ">>"), data in log.txt (if there is any) will be discarded before continuing with the following commands.

That is what the exec command above did.

The shell variable $? is the exit status of the previous command. It is being saved in the variable ec because I need to use the value in the next two commands.

This increments the value of the shell variable i as the last of the commands in the while loop:

while condition
do      commands
done

After incrementing i , [ -f cmd$i ] (the condition in this while loop) tests whether or not there is a regular file named cmdx where x is the current value of the shell variable i ? If cmdx does exist, the commands in the loop are executed again with the updated value of i .