Script to transfer files without xcom

Hi All,

I want to make a script in which I want to transfer files from current server to another.
The problem is that I don't have xcom installed on this so I was thinking of using scp command.
The code I've written till now is -

# Transferring latest inactive log file to another server
echo "Checking if there are inactive log files on the home directory...."
HOME_DIR=/o2/logs/weblogic/o2prl/
if [ `ls ${HOME_DIR}o2prl-05.log0* |wc -l` ];
then
echo "Inactive files are present"
echo "Continuing to further check the latest inactive file"
# Counting the number of inactive files
filename=`ls ${HOME_DIR}o2prl-05.log0* | tail -1`
file=`basename $filename`
# Start file transfer
scp ${HOME_DIR}${file} n412426@blx07ap09:/o2/home/
else
echo "There are currently no inactive log files on the home directory...."
fi

This is not running, basically I want to transfer file without xcom.
Please suggest.
Thanks

You have lots of options including ftp, remsh/rsh, sftp, NFS, samba and probably more. It depends what you are trying to acheive overall, if your servers are local, is the data sensitive (e.g. client personal details or company secret values)

Can you elaborate a little to stop us shooting off down an unacceptable path?

What are your OS versions for the two sides?

Robin
Liverpool/Blackburn
UK

1 Like

Thanks for the reply Robin.
The OS versions of the two sides are -
GNU/Linux 2.6.18-194.3.1.el5
The data that I need to transfer is sensitive and the servers are not local and they have super user accounts too.

and what you want to achieve at the end? between those two boxes?

Probably better to avoid sending in files with a superuser account if you can avoid it. Can you create a standard account that has write permission to the required area on the target server? It would be a little safer.

After that you are probably better with SFTP. Would a plain FTP (of a similar sized file) transfer in a suitable time? You say that it is sensitive, but this is just to get an idea of throughput and what is acceptable. If we can avoid compression, that saves a bit of work.

Other options depend on the tools you have available and the network access, for instance, do you have an encryption tool such as PGP? We could use this with rsh/remsh to copy the data.

It's all a bit of trying to fully understand the requirements at first along with options, limits and potential problems. I have a few ways in my head, so it's trying to pick the best avenue.

What more can you tell us?

Robin
Liverpool/Blackburn
UK

As per the requirement, compression not required as such.
I tried using ftp as -

# Transferring latest inactive log file to another server
echo "Checking if there are inactive log files on the home directory...."
HOME_DIR=/o2/logs/weblogic/o2prl/
HOST=blx07ap09
USER=o2prlods
PASSWD=xxxxxxx
if [ `ls ${HOME_DIR}o2prl-05.log0* |wc -l` ];
then
echo "Inactive files are present"
echo "Continuing to further check the latest inactive file"
# Counting the number of inactive files
filename=`ls ${HOME_DIR}o2prl-05.log0* | tail -1`
file=`basename $filename`
# Start file transfer
ftp -i -n $HOST
user ${USER} ${PASSWD}
cd ${HOME_DIR}
put ${HOME_DIR}$file
else
echo "There are currently no inactive log files on the home directory...."
fi

However as I am new to this area, hence, this is not working.

---------- Post updated at 04:57 PM ---------- Previous update was at 04:49 PM ----------

I want to transfer file from the server I am making script in to the server blx07ap09.

Be aware that a plain FTP will pass your data accross the network in readable form if someone can read in the network packets. Not a good idea to test with live data in any case, but also your credentials are passed in clear text. If you are using a superuser account, then you can easily be comprimised :eek:

The problem with your script is that when FTP starts, it has no input defined, so it will default to the screen. You would need to do something like:-

# Start file transfer
ftp -i -n $HOST  <<-EOFTP
user ${USER} ${PASSWD}
cd ${HOME_DIR}
put ${HOME_DIR}$file
EOFTP
else
......

... to get it to run.

We should be using SFTP for this, but it is set up and called in a different way. Hopefully the steps are as follows (please correct me anyone else that sees this)

  1. Create standard user account on target server with write permission to the required area
  2. Login in as the new account and generate a pair of encryption keys
  3. FTP the public key to the source server
  4. Add the new account to /etc/ftpusers to prevent normal FTP connections
  5. Test the SFTP connection

So, tackling these:-

  1. I'm assuming that this is okay.
  2. Generating keys is accomplished with the command ssh-keygen if you have installed the SSL software. Just run it on the command line and press ENTER to the prompts of where to store it and use a blank passphrase.
  3. I assume you can acheive this. The file to copy is .ssh/id_rsa.pub in the home directory of the new user. Put it on the source server as .ssh/authorized_keys ensuring the directory is rwx --- ---
  4. Add the account with the editor of your choice.
  5. Run sftp $targethost. It will ask you to verify the target firsdt time. You should see either an encryption key fingerprint string or picture. Accept this and it will create a record in .ssh/known_hosts allowing access next time, but warning you if it changes. Close the SFTP with the usual bye quit or close for a normal FTP.

If you get all that running, we can carry on with automating it, but best to get this far first. Then others can chip in too. I do not proclaim to be an expert in this (or any other) area.

I hope that this helps to start off with. Do ask for help if you want it and I haven't explained properly.

Robin
Liverpool/Blackburn
UK

1 Like

Hi Robin,

Some more things I want to clarify (I tried googling too but it got me more confused).
1) - I want to transfer file from location /o2/logs/weblogic/o2prl/ from server blx28ap05.
2) - My shell script is in location /o2/home/n412426 on blx28ap05
3) - I want to transfer file to server blx07ap09 on location /o2/home.

I want to try ftp first (as I am just testing this now) after that I'll try sftp when I'll prepare this for live data.

Okay, well first we should check that /o2/home/n412426 is on your search path:-
echo $PATH
The output will be a list of directories separated by colons. If the above is your home/default directory then it may well be in the list already. You can extend the search list with:-

export PATH=$PATH:/o2/home/n412426

or

export PATH=$PATH:$HOME

... if it is your home directory.
Assuming that you have write permission to /o2/home on blx07ap09, then the following should get you going:-

ftp -n blx07ap09 <<-EOFTP > /tmp/myftp.log
user $userid
$password
cd /o2/home
put $filename
EOFTP
echo "Finished FTP with return code $?"

You may need to adjust this by putting the password to the end of the previous line. The EOFTP must only be indented by tab characters, not spaces. The output will be caught in /tmp/myftp.log

You can change the target filename by appending it to the end of the put statement if you wish.

I hope that this gets you going enough for now. If it doesn't work, could you post back the messages it returns.

Robin
Liverpool/Blackburn
UK

Hi Robin,

I amended the script as you suggested.
But now when I do "./script_name", I only get the following -

Checking if there are inactive log files on the home directory....
Inactive files are present
Continuing to further check the latest inactive file
Latest inactive file is o2prl-05.log00427
Starting transfer of file o2prl-05.log00427

and then nothing and when I press cntrl+C, it gives output as -

Finished FTP with return code 0"

The file that is currently to be transferred is -

-rw-r--r-- 1 o2prladm o2     5634 May  8 16:22 o2prl-05.log00427

Seems like I am making script using my ID (n412426) and the file that needs to be transferred is in o2prladm user.
I've checked and verified that the destination directory has write access to my account n412426.

Thanks and Regards,
Rohit

Perhaps you need a quit before the EOFTP. Did you get anything in the log file? :confused:

Robin

There is nothing in the log file..it is empty..:confused:
I've put quit before EOFTP and now the whole script looks like -
:wall:

# Transferring latest inactive log file to another server
echo "Checking if there are inactive log files on the home directory...."
HOME_DIR=/o2/logs/weblogic/o2prl/
HOST=blx19au01
USER=n412426
PASSWD=xxxxxx
if [ `ls ${HOME_DIR}o2prl-05.log0* |wc -l` ];
then
echo "Inactive files are present"
echo "Continuing to further check the latest inactive file"
# Counting the number of inactive files
filename=`ls ${HOME_DIR}o2prl-05.log0* | tail -1`
file=`basename $filename`
echo "Latest inactive file is $file"
# Start file transfer
echo "Starting transfer of file $file"
ftp -n $HOST <<-EOFTP > /o2/home/n412426/myftp.log
user ${USER}
${PASSWD}
cd /o2/home/n412426/
put $file
quit
EOFTP
echo "Finished FTP with return code $?"
else
echo "There are currently no inactive log files on the home directory...."
fi

:wall:

NOTE : I've changes host to blx19au01 as my ID (n412426) has access to my home directory (/o2/home/n412426).

I would therefore assume that it's waiting for the password to be typed. Try putting it on the same list as the userid:-

...
...
ftp -n $HOST <<-EOFTP > /o2/home/n412426/myftp.log
user ${USER}  ${PASSWD}
cd /o2/home/n412426/
...
...

... and see if that helps.

Robin

No luck still.
Whe I waited for script to complete...after 4-5 misn it gave the following output -

Checking if there are inactive log files on the home directory....
Inactive files are present
Continuing to further check the latest inactive file
Latest inactive file is o2prl-05.log00427
Starting transfer of file o2prl-05.log00427
ftp: connect: Connection timed out
Finished FTP with return code 0

Okay, let's back away from scripting it for a moment. Can you try to run FTP from the command line and show the output (obviously obscuring anything sensitive)
I'm hoping you will get something like:-

$ ftp myserver
Connected to myserver.mycomp.co.uk.
220 MYSERVER FTP server (Version 4.1 Sat Jan 27 11:34:22 CST 2001) ready.
Name (myserver:RBATTE1): RBATTE1
331 Password required for RBATTE1.
Password: 
230 User RBATTE1 logged in.
ftp>

Does this sort of match? If it fails to connect, then it could be a firewall issue. If it opens the connection (login prompt) but rejects you, then it could be userid/password, an entry for your id in /etc/ftpusers or some other locking/security mechanism in play.

Robin

What about creating sshkeys and then using scp or rsync?

Read the thread. We're getting to that (I hope)

Robin