Script working on command line and not on crontab

my problem is this: when I run a script from the command line it works but returns a failure if I run it from crontab.
Basically I wanted to send a file to hdfs,
I thought it was related to the fact that crontab do not know the path to hdfs so I put the full path but it still does not work: here is the piece of code that fails :

/usr/bin/hdfs dfs -put -f  $source_directory/$file $hdfs_target_directory

and here is the nice error message

put: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "aocsv155bed0p.foo.acme.net/10.12.11.133"; destination host is: "aocsv155bna1p.foo.acme.net":8020; 

Any help please ? :rolleyes:

Your error says, no valid credential provided.

How to fix it ?

Check below link:

Kerberos Authentication on Hadoop Cluster - Stack Overflow

Step 16: Verify that Kerberos Security is Working | 5.5.x | Cloudera Documentation

thank for the links sadique
The script works when I launch it on the command line and I don't have this credential problem , the same user ( who is me ) when I try to run it on crontab I get this error !!!

how are you running in crontab?

please share the part of code.

Here is the code :

for file in $FILE_LIST
       
 do
                /usr/bin/hdfs dfs -put -f $source_directory/$file $hdfs_target_directory
                res=$?
                       if test "$res" != "0"; then
                            echo "the hdfs put command failed with: $res"
                       else
                            echo "the hdfs put command success with : $res"
                       fi
        done

Your problem seems to look like with java path.

is it autodirect or you have exported explicitly ?

Check this file:

more /usr/bin/hdfs

I believe cron jobs start with a reduced environment. I would guess that you either have "logged in" to something prior to running the script, or there is a setting in your .profile or .bashrc file that allows your script to be run from the terminal. This probably won't work, but try prefixing your command line (within the crontab) with /usr/bin/env (example):

30 40 * * * /usr/bin/env myscript arg1 arg2

Andrew

try below:

hdfs dfs -put -f $file $hdfs_target_directory

more /usr/bin/hdfs returns :

#!/bin/bash
  # Reference: http://stackoverflow.com/questions/59895/can-a-bash-script-tell-what-directory-its-stored-in
  SOURCE="${BASH_SOURCE[0]}"
  BIN_DIR="$( dirname "$SOURCE" )"
  while [ -h "$SOURCE" ]
  do
    SOURCE="$(readlink "$SOURCE")"
    [[ $SOURCE != /* ]] && SOURCE="$DIR/$SOURCE"
    BIN_DIR="$( cd -P "$( dirname "$SOURCE"  )" && pwd )"
  done
  BIN_DIR="$( cd -P "$( dirname "$SOURCE" )" && pwd )"
  LIB_DIR=$BIN_DIR/../lib

# Autodetect JAVA_HOME if not defined
. $LIB_DIR/bigtop-utils/bigtop-detect-javahome

export HADOOP_LIBEXEC_DIR=//$LIB_DIR/hadoop/libexec

exec $LIB_DIR/hadoop-hdfs/bin/hdfs "$@"

---------- Post updated at 08:23 AM ---------- Previous update was at 08:12 AM ----------

Even when I add /usr/bin/env it doesn't work.

To use Hadoop command you need to use kinit command to get a Kerberos ticket first, this kinit is generated automatically for me ( in the command line) but krontab ignores this .
My question is how to inform krontab so it considers this ticket ??

I don't understand what you meant by the above (in red). Do you run kinit on the command line?

As I said earlier, your interactive environment and your cron environment are different. Your active tickets will not be available in the cron environment.

I think you need to save the kerberos ticket in such a way your script can pick it up, using kinit (which is not present in your script). I think your best bet is to google for how to do this.

Andrew

@Andrew

"I think you need to save the kerberos ticket in such a way your script can pick it up, using kinit"

I think that if I can do that , I resolve the problem , but how can I do ?? :rolleyes:

Also try sourcing your .profile in crontab:-

30 40 * * * . ~/.profile 1>/dev/null 2>/dev/null; myscript arg1 arg2

@YODA

I did

I am not familiar with this technology, but I did some reading online and found few threads which might help you:-

Setting up cron jobs

Creating Keytab

Kerberos ticket error in cron job

As I said, time to google for it. It looks like you need

kinit -k -t /path/to/tab/file

in your script but I can't see how you create the file and I'm sorry to say I'm not prepared to waste any more time on this.

Andrew