queue items...

Hi everyone.

I have a lot of programs i want to run on some data files, they need to be done sequentially. Often the output from one program is the input for the next.

e.g

$ progA data1 > data1.A
$ progB data1.A > data1.AB
$ progC data1.AB > data1.ABC

repeat on data2, 3, 4, 5, 6 etc

bash seems to limit the number of times I issue commands whilst it is running the first. so the question: Is there any smart way to queue commands up, or would a simple script be better (I really am a beginner and have no idea how to do this)?

Thanks a lot, J

You've hit the nail straight on -- a simple script would be perfect. Not only does it allow you the freedom to run one command to accomplish several tasks, its easy to repeat the process if that is needed.

Excuse me if you know this, but on the off chance that you dont....
Edit a file with vi, or similar editor, add your commands in the order you desire, and save the file. You'll need to make the file executable, and once that is done you can run your script.

#!/usr/env/bin/ bash
set -e
command1 input-file >file1
command2 <file1 >file2
command3 <file2 >output

To make it executable, use the chmod command (assuming the filename is myscript.bash):

chmod 755 myscript.bash

Then you can run it by typing 'myscript.bash' on the command line.

The 'set -e' command causes the script to exit immediately on error; prevents command2 and command3 from being executed if command1 fails. You can do your own error checking, and maybe recovery, but the 'set -e' is the easiest. If these commands are long running, I like having an echo message in the script letting me know what is currently running:

#!/usr/env/bin/ bash
set -e
echo "$(date) starting c1" >&2
command1 input-file >file1
echo "$(date) starting c2" >&2
command2 <file1 >file2
echo "$(date) starting c3" >&2
command3 <file2 >output
echo "$(date) script complete"

had to change

#!/usr/env/bin/ bash

to

#!/bin/bash

but worked great thanks.

how could I change the script so I can input the name of x on the command line?

myscript.bash

input-file >file_x_1
file_x_1 >file_x_2

and so on..

Thanks very much, J

Oh how embarassing!! That should have been

#!/usr/bin/env bash

Guess I've taken fatfingering to a new level

Use $1 instead of x in your filename.

file_${1}_1

You'll need to be careful as the underbar (_) character is treated as a part of a variable name, so put the variable name in {} to keep things right. In this case $1 is the first argument on the command line after the command name. So if you entered `myscript foo` your filenames would be `file_foo_1` and so on. You might also want to test to ensure that a parameter was entered on the command line:

if [[ -z $1 ]]
then
    echo "missing command line parameter"
    exit 1
fi

The test causes the error message and immediate exit from the script if the variable $1 is empty (not entered on the command line).

thanks, this is so cool. I can write a script and leave my analysis running overnight now. Thanks again. J