Script containing Curl doesn't run with crontab

Shell Programming and Scripting
1

Hello
I have a problem with the crontab command
when I run a code containing Curl on the command line it runs without fail but as soon as I program it with crontab it executes everything except the curl returns fail
thank you for helping me to resolve this problem because since Monday I look without finding an issue

ps. I got this message: curl: (7) Failed connect to apirest.atinternet-solutions.com:443; Connection timed out

thank you in advance

2

This is most likely because the environment for the process set up by cron is very basic. You may need to include all sorts of things such as proxy variables in your called script.

Can you share the code and details of what OS/version you are using? Sanitised output from uname -a should be fine.

Kind regards,
Robin

1 Like
3

Thank you Robin for your reply,
uname -a returns :

Linux aocsv155bed0p.cloudwatt.saint-gobain.net 3.10.0-327.36.3.el7.x86_64 #1 SMP Mon Oct 24 16:09:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

and this piece of code returns fail thoug it works fine in th command line :

if /usr/bin/curl -LS -D- -X GET -H "Authorization:Basic $password" -H 'Content-Type: text/csv' -L -o  $OUTPUT_FILENAME $url; then
                       # if wget GET password=$password -H 'Content-Type: text/csv' password=$password  -o  $OUTPUT_FILENAME $url; then
                            echo "success"
                        else
                            echo "fail"
                        fi
4

What be the contents ot the variables password , OUTPUT_FILENAME , url when run from cron ?

1 Like
5

What are your values for the following variables in your interactive shell and what are they set to in your script?

http_proxy
HTTPS_PROXY
FTP_PROXY
ALL_PROXY
NO_PROXY

Thanks again,
Robin

6

excuse me Robin , I'm beginner in shell so I don't how to get theses informations http_proxy etc

7

put

echo $http_proxy
echo $HTTPS_PROXY
echo $FTP_PROXY
echo $ALL_PROXY
echo $NO_PROXY

just in front of the curl command.

8

for ftp , all and no proxy : no display

when I run the script on the command line I get the two addresses for http proxy and https proxy and nothing else

proxy.speria.saint-trop.net:3128 
proxy.speria.saint-trop.net:3128

but when I run it with crontab I get nothing at all

9

Give it a try setting the two in your cron script.

10

how can I do that plz ?

11

Try

http_proxy="proxy.speria.saint-trop.net:3128"
HTTPS_PROXY="proxy.speria.saint-trop.net:3128"
1 Like
12

Without these variables, the cron-run script will not know how to get out of your network. Why would it? These are probably in a profile for you somewhere.

You can explicitly set them in your cron-run script, or if it might be called from the command line and people may want to use their own, you could:-

http_proxy="${http_proxy:-proxy.speria.saint-trop.net:3128}"

This will use the value proxy.speria.saint-trop.net:3128 for http_proxy if the variable has not been set yet, i.e. this is a default but an existing setting will be retained.

Obviously do the same thing for the https_proxy variable too.

I hope that this helps,
Robin

1 Like
13

Thank you guys , I'll test this solution

---------- Post updated at 09:57 AM ---------- Previous update was at 09:13 AM ----------

I got this error

0curl: (5) Could not resolve proxy: ${https_proxy; Name or service not known

Is this the right syntax :

 http_proxy="${http_proxy:-proxy.speria.saint-trop.net:3128}"

??

---------- Post updated at 10:21 AM ---------- Previous update was at 09:57 AM ----------

Thank you sooooooo much guys , it works :slight_smile:

14

If it's working, would you care to share the completed solution (or the relevant bits) for anyone finding this script with a similar problem in future?

Thanks, in advance,
Robin

15

But there's an other problem,
when I get the csv file I try to put it via hdfs but the put fails
and I get this message :

Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed

---------- Post updated at 11:19 AM ---------- Previous update was at 11:13 AM ----------

Yes of course Robin ,

I just added the two links in the cron-run script :

http_proxy="proxy.speria.saint-trop.net:3128"
HTTPS_PROXY="proxy.speria.saint-trop.net:3128"

As you mentioned in a post : "the cron-run script will not know how to get out of your network"

Hope that this helps