Job runs manually, doesn't work in crontab

I have a script (/home/admin/run_bkup.sh) that I can run manually to kick off an executable job. I want to run it in crontab, but it doesn't work.

Here's the script:

shell=/bin/bash
today=$(date +"%m-%d-%y")
/opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export mgt-svr-bkup-$today << EOF
y
n
EOF

It would be easier to just add the executable job to crontab, but the executable job is interactive, requiring a response to two questions.

Here is the crontab entry:

00  6  *  *  1   /home/admin/run_bkup.sh

What do I need to do to get this job to run in crontab?

Try specifying full paths for all the commands in the script.

/opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export is the full path.

Source the user environment before executing in cron.
Check the $HOME directory for environment files and/or env command.

Also the syntax defines shell variable as /bin/bash for some usage.
I do not see it being used further in the program, but one can only the guess what is upgrade_export , perhaps it is used in it.

You would want to define a magic cookie in your script !#/bin/bash as the first line.

Further on, i believe that export can be done from managment of checkpoint fw or ?
You might want to notice the post under this thread, since people are having these types of issues all the time.

date is not full path. If you told us WHAT "doesn't work" you might get a more meaningful answer - if the references at the bottom of this page don't suffice.

mgt-svr-bkup-$today is not an absolute pathname either. There is no reason to believe that your cron job is going to run in the directory in which you invoked the crontab utility to add your cron job to the schedule.

• Thx, Peasant, but I don't know what this means.

• RudiC, thanks for the response. There is no output that I can see.

• Don, thx for the response. I didn't think

mgt-svr-bkup-$today

was a file that required an absolute path. The output file from the executable job is

mgt-svr-bkup-$today.tgz

. I assumed

mgt-svr-bkup-$today

was a parameter passed to the job to be used when creating the output file. If I'm wrong, and it is the file (without the .tgz extension), then this should work? -

shell=/bin/bash
today=$(date +"%m-%d-%y")
/opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export /opt/CPsuite-R77/fw1/bin/upgrade_tools/mgt-svr-bkup-$today << EOF
y
n
EOF

Without you showing us the contents of /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export (if it is a shell script) or the source code that was used to produce /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export (if it is a binary executable) we have absolutely no way to guess at what the argument to it means. It that parameter is used to create an output file, maybe /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export did create that file for you somewhere (in a directory you haven't searched where cron ran your job). If that parameter names an input file, you absolutely either need to be sure that your script changes its current working directory to that directory before it starts /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export or you need to provide the name of that file as an absolute pathname.

The common thing about the hundreds of threads started in this forum about a job that works when run on the submitter's shell command line works, but doesn't work when run from cron is that cron does not store your current shell execution environment in crontab and setup that same shell execution environment for the job when it is run. YOU have to make sure that any script you run from cron creates a shell execution environment that uses the proper shell to run your cron job, positions itself in a directory where your script has access to input and output files it needs, and has environment variables set appropriately for any variables the processes started by your script need to operate properly.

Your request is essentially saying my shell script doesn't work. Fix it for me. And, without any details about what your script does, you KNOW that we can't possibly do that other than to make the generic suggestions we have already given you.

Thx again for the response. I don't know how to show the source code for the executable. How do I do that?

I do know that the job is NOT creating the output file anywhere.

"Your request is essentially saying my shell script doesn't work. Fix it for me." - That's not what I'm saying (or not my intent). The script does work. It runs the upgrade_export executable that creates an output file -

/opt/CPsuite-R77/fw1/bin/upgrade_tools/mgt-svr-bkup-$today.tgz

So I'm not asking to fix the script, I'm wondering why it runs manually and not in crontab. If the reason is I need to set the correct shell execution environment, then at least I have something to work on. But my thought in posting this was maybe there was something obvious I was missing that was the problem. I'm new at this and trying to figure it out.

If you want a simple fix - start with environment variables.

There are two commands: set and env . set is a shell builtin and shows all variables including those visible locally (not exported). env command cannot do that, it sees only those environment variables the process got at process creation. env is a separate physical file not a builtin. So you want to use env to really see what is going on -in your case not going on.

One crude way:
create a one line script called script.shl:

/usr/bin/env

run it two ways -- (redirect the output of each run to different file names. For crontab /tmp is a handy place) manually, then in crontab

The diff command will see what crontab does to your login variables. Add 'em to your script that runs the problem executable - be sure to export the variables.

To post code use code tags -- these are examples with embedded spaces; html does not interpret them
[ c o d e ]
script text goes here
or FORTRAN, C, C++ ,etc. code goes here between a separate set of code tags for each program file.
[ / c o d e ]

Obviously, remove the spaces from the example to make it work.

script text goes here
or FORTRAN, C, C++ ,etc. code goes here between a separate set of code tags for each program file.

Look at the threads at the bottom of this page. There are five threads there with exactly the same problem you are having and the same suggestions we have been making here solved the problem for all of them.

When you run the program manually and it works, what shell are you using? Change the 1st line of your script from:

shell=/bin/bash

to:

#!pathname

where pathname is the absolute pathname of the shell you are using. Does that change make your script work when run by cron ?

If not, when you run the program manually and it works, what directory is your current working directory? What is the output from the command:

pwd -P

? Add a command to your script before you invoke /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export that changes your current working directory in your script to that directory:

cd directory

where directory is the string returned by the above pwd command. After that change, does your script work when run by cron ?

When you run the program manually and it works, what environment variables does your program need to run correctly? Does your program work correctly if you invoke it manually with the commands:

today=$(date +"%m-%d-%y")
env - PATH="/bin:/usr/bin" /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export mgt-svr-bkup-$today << EOF
y
n
EOF

? If not, what environment variables does /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export need to run correctly? Find out (using the methods suggested by Jim McNamara) and set those environment variables in your script before you invoke /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export .

Thanks, Jim!
I created the script /usr/bin/env and ran it manually and in crontab .

The PATH variable in the manually-run output file is long. The PATH variable in the file run in crontab is short - PATH=/usr/bin:/bin. So, thinking I needed to tell cron to use the long PATH variable, I added it into the script: (PATH=/opt/CPsuite-R77/...), but that didn't work when I ran the job from crontab.

Thanks for the response, Don.
"Look at the threads at the bottom of this page. There are five threads there with exactly the same problem you are having and the same suggestions we have been making here solved the problem for all of them."

If that were true, I'd have figured it out by now. The problems in the other threads are "similar", but not "exactly the same". I read through them before I posted my issue. Had I been able to solve my problem using suggestions posted in the other threads, I wouldn't have posted mine.

I get that my problem probably has to do with setting the environment variables, but it's not clear to me yet how to do that.

"When you run the program manually and it works, what shell are you using? Change the 1st line of your script from:

shell=/bin/bash

to:

#!pathname

where pathname is the absolute pathname of the shell you are using. Does that change make your script work when run by cron ?"

Since the first line in the script is shell=bin/bash, I'm using the bash shell, right? Changed the first line to #!/bin/bash. Still doesn't work using cron.

I'm running the job from my home directory -

/home/admin

"? Add a command to your script before you invoke

/opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export

that changes your current working directory in your script to that directory:

cd directory

where directory is the string returned by the above pwd command. After that change, does your script work when run by cron ?"

Still doesn't work.

"When you run the program manually and it works, what environment variables does your program need to run correctly? Does your program work correctly if you invoke it manually with the commands:"

today=$(date +"%m-%d-%y")
env - PATH="/bin:/usr/bin" /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export mgt-svr-bkup-$today << EOF
y
n
EOF

Yes.

Yes, No, Maybe!
If the default shell for your system is /bin/bash , then having the line:

shell=/bin/bash

in your script does nothing and your script is being run by bash . If the default shell for your system is NOT /bin/bash , then having the line:

shell=/bin/bash

in your script does nothing and your script is NOT being run by bash ; it is being run by the default shell on your system.

If you type the command:

shell=/dev/null

into your shell, does the shell you are using magically change to /dev/null ? NO. All it does is set a shell variable that later lines in your script (or, when done interactively, in your login session) can access by testing the value of $shell .

By convention (when your login shell is not a csh derivative), when you login the shell variable SHELL is exported and set by the system to an absolute pathname of your login shell. My login shell is /bin/ksh and after I login, $SHELL expands to /bin/ksh . If I then invoke bash , $SHELL still expands to /bin/ksh . Note also that case matters: $SHELL and $shell are two different variables.

If the script you are asking cron to run is an executable file and the 1st two characters in that file are #! , the system will use the interpreter named by the pathname on the rest of that line as the program used to interpret the rest of that file.

Now, back to your environment problem...
If you don't know what environment variables /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export needs to have set in order for it to work correctly, you need to set all of the variables in your script to the same values that they have when you run /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export manually and it works.

When you login with bash as your login shell, it runs three files from your home directory if they are present: .bash_profile , .bash_login , and .profile . If those files are where the variables /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export needs to run are defined, the following might work for you. Change:

shell=/bin/bash

in your script to:

#!/bin/bash
HOME="/absolute/path/to/your/home/directory"
cd "$HOME"
[ -f .bash_profile ] && . .bash_profile
[ -f .bash_login ] && . .bash_login
[ -f .profile ] && . .profile

One would expect that one or more of those three files initializes PATH (although it appears that your script doesn't depend on any utilities that are not on the default PATH that you aren't referencing with absolute pathnames) and other shell variables to the values you'll need to run /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export , but we can't know that for sure since we have no idea what other commands you have run after you last logged in to your system before you successfully run /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export from your interactive shell. And, as we have repeatedly said, without knowing what /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export is doing, we are just guessing. If /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export tries to open your controlling terminal for some reason, it might just not be possible to run /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_export from cron at all.

Thanks for the explanation, Don.

I changed

shell=/bin/bash

in the script to

#!/bin/bash
HOME="/home/admin"
cd "$HOME"
[ -f .bash_profile ] && . .bash_profile
[ -f .bash_login ] && . .bash_login
[ -f .profile ] && . .profile

Still didn't work using cron.

I don't need to run any commands after logging in before manually running the job for it to work.

How can I tell what an executable job is doing? I know it would help solve this if I had that information, but I don't know how to check, since it isn't a text file.

You don' t need to run any of those commands manually after logging in because logging in runs those commands automatically before you see your first shell prompt. But, when you run a job with cron you don't login and your script has to do whatever it takes to create an environment in which the processes started by your script running under cron have the environment they need to get their work done.

As Jim McNamara suggested, create the following script in a file named get_cron_env :

#!/bin/bash
HOME="/home/admin"
cd "$HOME"
[ -f .bash_profile ] && . .bash_profile
[ -f .bash_login ] && . .bash_login
[ -f .profile ] && . .profile
env > environ.cron

Make that file executable with:

chmod +x get_cron_env

Set a crontab entry to run that script (using an absolute pathname for the script; not just get_cron_env ).
And then run the commands:

env > /home/admin/environ.interactive
diff /home/admin/environ.cron /home/admin/environ.interactive

and show us the output.

There will likely be some differences because interactive environments have more stuff going on than background jobs like those started by cron . But, maybe we'll get lucky and see something that you can easily fix.

I can't help you find the source for the executable you have installed as /opt/CPsuite-R77/fw1/bin/upgrade_tools/upgrade_expor . If you (the royal you meaning some administrator inside your company) wrote software and didn't provide documentation on how to run it, track down who wrote it and get them to tell you exactly what you have to do to run it from cron .

If this is third party software you bought from someone else, read the documentation that came with it. If you bought code from someone and didn't get documentation that tells you how to run it; shame on you. Contact the vendor and ask for support.

If this is third party software you downloaded free from someone else, read the documentation that came with it. If you downloaded code without documentation and without source, and without support; you've got what you paid for and I wish you luck. If you downloaded free source, look at it and figure out what it is doing that keeps it from working under cron and fix it. Or send mail to the support site for that free software and ask for help.

And, of course, if you downloaded a bootlegged copy of someone's software from a black site, we have absolutely no sympathy for you and will not further assist you in trying to fix code you stole from its creators. (But, we assume that you would never do that.)

Thx, Don!

Sorry, too busy today to get to this. I'll create the script tomorrow, run the

DIFF

command, and post it here.

I inherited this without a lot a documentation, and I've been learning on the job. The code is legitimate and comes from Check Point Software Technologies (hence CPsuite-R77). I don't have any doc on it, but I'll try to find out more about it tomorrow.

What prompted this is the network team here at work was running this executable manually every Monday morning. I'm trying to automate it so we don't have to do that.

---------- Post updated 06-17-16 at 09:35 AM ---------- Previous update was 06-16-16 at 06:01 PM ----------

Here is the output from the command

diff /home/admin/environ.cron /home/admin/environ.interactive

2a3,4
> HOSTNAME=admin-comm
> TERM=xterm
4c6,7
< MAILTO=
---
>HISTSIZE=1000
> SSH_CLIENT=114.18.7.232 60699 22
6c9
< OLDPWD=/home/admin
---
> SSH_TTY=/dev/pts/2
19c22,23
< PATH=/opt/CPinfo-10/bin:/usr/bin:/bin:/opt/CPshrd-R77/bin:/opt/CPshrd-R77/util:/opt/CPshrd-R77/web/Apache/2.2.0/bin:/opt/CPsuite-R77/fw1/bin:/opt/CPsuite-R77/fw1/oracle_oi/sdk:/opt/CPsuite-R77/fg1/bin:/opt/CPda/bin:/opt/CPportal-R77/webis/bin:/opt/CPportal-R77/portal/bin:/opt/CPrt-R77/bin:/opt/CPuepm-R77/bin:/opt/CPuepm-R77/engine/jre/bin:/opt/CPshrd-R77/database/postgresql/bin:/opt/CPSmartLog-R77/bin:/home/admin/bin
---
> MAIL=/var/spool/mail/admin
> PATH=/opt/CPinfo-10/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/opt/CPshrd-R77/bin:/opt/CPshrd-R77/util:/opt/CPshrd-R77/web/Apache/2.2.0/bin:/opt/CPsuite-R77/fw1/bin:/opt/CPsuite-R77/fw1/oracle_oi/sdk:/opt/CPsuite-R77/fg1/bin:/opt/CPda/bin:/opt/CPportal-R77/webis/bin:/opt/CPportal-R77/portal/bin:/opt/CPrt-R77/bin:/opt/CPuepm-R77/bin:/opt/CPuepm-R77/engine/jre/bin:/opt/CPshrd-R77/database/postgresql/bin:/opt/CPSmartLog-R77/bin:/home/admin/bin
20a25
> INPUTRC=/etc/inputrc
27a33
> SHLVL=1
29d34
< SHLVL=2
33a39
>SSH_CONNECTION=114.18.7.232.60699 114.18.45.8.22
39c45
< _=/usr/bin/env
---
> _=/bin/env

Hoping for a response on this.

The input shows us that you are manually changing PATH significantly after you login before you successfully invoke your script manually (which I find surprising), but since the your script works manually when PATH is just set to /bin:/usr/bin that probably isn't your problem. Nonetheless, since the number and order of directories before /bin and /usr/bin in PATH and even the order of these two directories changed between your cron job environment and your manual invocation environment, I would suggest that you explicitly set PATH in your script just before you invoke the command that is failing.

Nothing else in the differences between the environment variable setting raise any red flags, but with seeing the source code for the command that is not working we are just guessing. You NEED to contact the manufacturers of that code and ask them what you need to do to run their code from a cron job.