I am writing a script to automate FTP files from AIX to Mainframe. ( Binary mode, passive connection)
FTP the currently available files(poll a directory to find the list) and put it to mainframe in one connection instance
Verify if all the files are copied to Mainframe successfully
If the FTP command fails due to some issue, try to resend the files and if the issue persist after 5 attempts to resend, inform particular user by email.
[list]
If the retry succeeds(send only those files that was not sent., this is in case of partially succeeded transfer; say 5 files , 2 transferred and command failed due to some issue and other 3 not sent... in that case send only those 3 files).
[/list]
Now again verification of file(s) transferred to mainframe
Below is the script that I have written so far
Issue/Doubt/Question/Help required:
Not able to do FTP with out giving password. Please let me know how to use .netrc here. Also please suggest any good alternative approach
I don't have any idea about verifying the transferred file to Mainframe
Requesting for help / guidance;
Pseudo code/ outline:
#! /usr/bin/ksh
########################################
#-- Defining Variable --#
########################################
DATE=`date +%Y%m%d`
POLL_DIR=/home/abcd/processing
TMP_DIR=/home/abcd/processing/tmpdir
LOG_DIR=/home/abcd/processing/log
KSH_REC=$POLL_DIR/`basename $0`"_rec" ##-- Recovery file for the script
# This will have current state of the script and will be used at while
# restarting the script after failure(before failing the job updated the rec file)
HOST=`hostname`
LOCK_FILE=`basename $0`_$DATE.lock
STOPPER_FILE=`basename $0`_$DATE.stop
MASTER_LOG_FILE=$LOG_DIR/ftp_to_mvs_$DATE.log
PROCESSED_FILES=$TMP_DIR/processed_files_$DATE.dat
SOURCE_SERVER=`hostname`
REMOTE_SERVER='remoteserver.abcd.com'
REMOTE_PORT=12345 ##-- Dummy, not sharing the details here in open forum.
REMOTE_USER=userid
REMOTE_USER_PASS=userpasswd
POLL_COUNTER=0
########################################
#-- Defining Functions --#
########################################
##--------------------------------------
#-- (1) function FTP to MVS
##--------------------------------------
f_FTP_MVS()
{
echo 'user $REMOTE_USER $REMOTE_USER_PASS' > $TMP_BATCH_FTP
echo 'type binary' >> $TMP_BATCH_FTP
echo 'passive' >> $TMP_BATCH_FTP
cat $TMP_BATCH_FILE >> $TMP_BATCH_FTP
echo 'bye' >> $TMP_BATCH_FTP
ftp -sinv $REMOTE_SERVER $REMOTE_PORT << `cat $TMP_BATCH_FTP`
}
##------------------------------------------------------
#-- (2) function to verify files transferred to MVS
##------------------------------------------------------
f_VERIFY_FTP_MVS()
{
##Use nlist or something to verify the transfer
}
if [[ ! -f $
[[ -f $PROCESSED_FILES ]] && rm -f $PROCESSED_FILES; touch $PROCESSED_FILES || touch $PROCESSED_FILES; ## Remove and touch if it exist
## on first run; if not touch it. This is one time activity
while [ 1 ] ### Future enhancement --> Include a parameter for Date and compare with system time,
# script should execute only for current day 00:00:01 hrs to 23:59:59 hrs else graceful exit;
# include STOPPER FILEs for convenient stoping by support team
do
DATE_TS=`date +%Y%m%d%H%M%S`
** TMP_LOG_FILE=$LOG_DIR/ld_rdy_$DATE_TS.log #--- Remove this file when processed
** TMP_FILE_ld_rdy=$TMP_DIR/ld_rdy_$DATE_TS.dat #--- Remove this file when processed
** TMP_BATCH_FILE=$TMP_DIR/ftp_batch_$DATE_TS.dat #--- Remove this file when processed
** TMP_BATCH_FTP=$TMP_DIR/ld_rdy_$DATE_TS.ftp
FTP_RETRY_COUNTER=0
if [ `ls -lrt $POLL_DIR/*.ld_rdy | awk 'END {print NR}'` -gt 0 ] ##** Change the condition with 1. LS and 2. grep "not found"
then
### Creating the list of currently available load ready file(s) to process(which is to be FTP'ed to MVS
ls -lrt $POLL_DIR/*.ld_rdy | awk '{print $NF}' > $TMP_FILE_ld_rdy
awk '{print "put", $0}' $TMP_FILE_ld_rdy | sed 's/.ld_rdy//g' > $TMP_BATCH_FILE
###########################
### FTP To MVS Part ###
###########################
echo "POLL COUNTER\t:\t"$POLL_COUNTER > $TMP_LOG_FILE
echo "DATE TIME \t:\t"$DATE_TS >> $TMP_LOG_FILE
echo "Trying to FTP below file(s) to Mainframe: "$TARGET_SERVER >> $TMP_LOG_FILE
sed 's/.ld_rdy//g' $TMP_FILE_ld_rdy >> $TMP_LOG_FILE
f_FTP_MVS
### Check for success or failure of FTP command
if [[ $? -ne 0 ]]
then
### Statements following failed FTPs command
### Step 1: Update the log for failure and
echo "\nFailure in transfering file to targer server" >> $TMP_LOG_FILE
FTP_RETRY_COUNTER=$(($FTP_RETRY_COUNTER+1))
sleep 10;
### Step 2: Check if FTP connection can be established
## if not send an email to Support(if Counter is more than 2) stating there is FTP connection issue.
## ----- Create a Function and call it here.
### Step 3: If connection can be established then try re-sending the unprocessed file(S). -- Write a Function for this
## i. Create Processed and unprocessed file list
## ii. Try re-sending the unprocessed file
## iii. If succeeds., update the log and come out of this loop and start Polling
## iv. If fails, Update FTP_RETRY_COUNTER to ++1
## and Call the STEP 2 - Connection check Function if
## FTP_RETRY_COUNTER is less than or equal to 2
## v. If FTP failed again, send a note to support to check on the FTP connection
### Step 4: Verify the files transferred to MVS
echo "exit 1 ||
echo "POLL COUNTER\t:\t"$POLL_COUNTER > $TMP_LOG_FILE
echo "DATE TIME \t:\t"$DATE_TS >> $TMP_LOG_FILE
echo "File(s) not available in the polling directory at this moment.\nPOLLING DIR\t:\t"$POLL_DIR"\n" >> $TMP_LOG_FILE
else
echo "Below files were successfully transferred to target server"; `cat TMP_FILE_ld_rdy | sed 's/.ld_rdy//g'cd`
fi
else
echo "POLL COUNTER\t:\t"$POLL_COUNTER > $TMP_LOG_FILE
echo "DATE TIME \t:\t"$DATE_TS >> $TMP_LOG_FILE
echo "File(s) not available in the polling directory at this moment.\nPOLLING DIR\t:\t"$POLL_DIR"\n" >> $TMP_LOG_FILE
fi
done
There is an example part way down the page. The file should be readable only to the (process) owner much like SSH keys. The alternate is to use SFTP and SSH keys if the server supports it.
Do either of these help?
As for verifying, I think that SFTP includes that in its return code. For plain FTP, you could get the file back and check that the one sent matches the one that has made the return journey.
There's nothing wrong with using a .netrc to avoid the need to supply a ftp password in the script if you are comfortable with using ftp and its security risks in the first place.
For a .netrc file to work it MUST be (1) residing in the home directory of the user under which the script (or cron job) is running, and (2) MUST be owned by that user, and (3) MUST have access rights so that only that user can read/write the file (typically '600').
Records in the file MUST be formatted correctly like:
if the ftp nodename is fred, the userid is joe and the password is smith.
Then when ftp is called:
$ ftp fred
ftp will first lookup the node 'fred' in /etc/hosts (or any other resolution method available) to retrieve an ip address. If that is successful ftp looks for a .netrc file in the script users home directory. If it finds .netrc it will retrieve userid/password combination for 'fred' (if it exists) and use it to make the ftp connection.
It's dead easy to see if this is working. Login as the user and interactively issue the command:
$ ftp <nodename>
and if .netrc is configured correctly it will connect instantly without requesting userid, password or anything.
No such thing, passwords is how ftp works. Even the .netrc file contains passwords, though is safer than keeping them in your scripts themselves. It looks like this:
Those features deal with limitations in FTP's protocol, and aren't relevant anywhere else. You don't have those options, because sftp isn't ftp. You also don't need them for the same reason. sftp doesn't possess these FTP features at all, so you can consider it to be always "binary" and "passive".
Ok Sure. That was additional question on top of that.
This has resolved the first blocker.
Now how do I verify the transferred file to Mainframe; below are the scenarios:
FTP the currently available files(poll a directory to find the list) and put it to mainframe in one connection instance
Verify if all the files are copied to Mainframe successfully
If the FTP command fails due to some issue, try to resend the files and if the issue persist after 5 attempts to resend, inform particular user by email.
4.If the retry succeeds(send only those files that was not sent., this is in case of partially succeeded transfer; say 5 files , 2 transferred and command failed due to some issue and other 3 not sent... in that case send only those 3 files).
Now again verification of file(s) transferred to mainframe
I dont have any idea about how to verify., looking for your advise / discussion.
If you're using FTP, some FTP clients support ls -l , but it sounds like you need a lot more than that.
If sftp's an option, have you considered rsync? You can consider it to be "cp on steroids". It's an extremely common UNIX utility which connects with ssh (same protocol as sftp) and does a lot of what you're asking for, especially the verification of files.
i.e. you can do rsync -c localfile1 localfile2 localfile3 localfile4 user@hostname:/path/to/remotefolder and it will copy and verify. It will even skip already-existing files based on checksum.
It would make your task very easy.
exec 2>errorlog
N=0
while ! rsync -c localfile1 localfile2 localfile3 localfile4 user@hostname:/path/to/remotefolder
do
N=$((N+1))
if [ "$N" -ge 5 ]
then
echo "rsync couldn't transfer files, see errlog" >&2
exit 1
fi
done
echo "rsync successfully transferred files"
Thanks Much Corona688., appreciate your help.
I have to implement this functionality using FTP.
Ok., since you have given me the above code I would like to ask few questions.
A. Using FTP method :- Will it not be possible to verify the file transferred and can the restart ability of the script be not achievable?
If so can you please throw some light?
B. Is using SFTP less secure when compared to FTP -s to transfer files from AIX to Mainframe.? Will there be any loss in data .i.e., will the data get corrupted?
And per my understanding if we go with SFTP method, we can used rsync to copy as well as verify the file.
Even in case of failure, restartability is assured(i.e, only files that are not copied will be copied to the mainframe).
The nice thing about FTP - and also the problem with FTP - is that it has zero features.
Can you ask FTP if the file uploaded successfully? Nope.
Can you ask FTP for a file checksum or anything like that to verify it? Nope.
Can it tell you the most recent file? Nope.
Can you ask it if a file is there at all? Nope.
Can it tell you anything? Nope. Well, a directory listing. That's it.
So, you can depend on FTP being reliably unhelpful. But there's old-fashioned ways to trick it into doing what you want:
ftp <<EOF
cd /remote/incomplete
put localfilename
rename /remote/incomplete/uploadedfile /remote/complete/uploadedfile
This prevents partial uploads from appearing in /remote/complete. If the connection is broken early, the file will never get renamed.
Then you can get the directory listing from /remote/complete to see if the file you wanted is in it.
It will require brute force.
Connect with FTP, saving all output to a file.
Get directory listing.
Quit.
Process directory listing in script to remove all lines except the directory listing.
Check if the directory listing contains the file you uploaded.
That sort of thing.
Do you even have ftp -s? That's a very rare feature.
sftp is less likely to corrupt your data than FTP.
FTP's "binary" feature is only there because, in the beginning, ftp was expected to translate text files automatically (i.e. EBCDIC to ASCII) so the recipient could read them. FTP mangles files because it's trying to process all downloads as text! And it always does that unless you force it not to with binary! sftp just doesn't do that, at all, period. It transfers the file as-is, always.
Wow.. that's lot of learning from just few interaction..
This is what I understood.
Step 1. Open a Control channel & data channel.. put the file
Step 2. Create a nlist to get the info of files transferred (if any other way please let me know)
Step 3. Quit
Step 4. Compare the nlist output and the list of files that was supposed to be transferred. If not transfer only rest of the files.
Please advise if any discrepancy in above steps using FTP method
Using SFTP method:
U said its interactive, may I please request you to give me examples or example scripts or few links for example/ to learn.
Also tell me the steps here
More over for SFTP should we use only port 21 or the same control port used for FTP can be used to connect to Mainframe. (Asking this question because I am not able to connect to mainframe using sftp)
If your FTP server and client support nlist, that's the best way to get the file list. Not all of them do.
You've got the idea, I think.
Nobody said that, and I'm not sure what you mean by it.
sftp is not ftp. Period. Full stop. It is not related to FTP at all, and does not inter-operate with it. Trying to connect to your FTP server with sftp is as nonsensical as connecting to your mail server with it. It's just not going to work. There is no control channel, either. You just connect to your destination directly.
sftp is actually a use of ssh protocol -- secure shell, i.e. port 22.
As I said in my post#2, use of a .netrc file should avoid the need to supply a userid/passwd for the ftp server.
Yes, many times I have built a command file to input to ftp:
$ ftp 'ftpserver' < 'command file' 1> /dev/null
and it works fine. You have to be careful of how to trap errors though. You also need to ensure that the last line of your command file is 'quit' to ensure control is passed to the next line of your script when ftp operation is done.
When you say that it doesn't work; in what way????? Error message or what? Give us all a clue.
Do you need to use a non-standard ftp port? If not, why try and specify it?
The correct format for using a here document would be something more like:
f_FTP_MVS()
{
ftp -sinv $REMOTE_SERVER $REMOTE_PORT <<EOF
user $REMOTE_USER $REMOTE_USER_PASS
type binary
passive
$(cat $TMP_BATCH_FILE)
bye
EOF
}
But, the AIX ftp man page that I have access to doesn't include a -s option. And, if you're trying to use ftp to copy the content of the file named by the expansion of $TMP_BATCH_FILE from AIX to MVS, the correct way to do that would be something like: put $TMP_BATCH_FILE
not catting a binary file into a here document.
Alternatively, you could use simple redirection instead of a here-document:
And, of course, unless the FTP protocol includes character set translations when moving from AIX to MVS... since AIX works in ASCII/ISO 8859-1/UTF-8 and MVS works in EBCDIC, you may also need a step somewhere to convert your ASCII requests and file contents into EBCDIC. One possible way to do that if you need to is with dd conv=EBCDIC or dd conv=IBM depending on which variant of EBCDIC is used on your target MVS system.
Thanks much for your response.
The intention was not to bump posts.
I was able to connect to the MVS password less using .netrc and give the command to put files into mvs.
Whereas when i create a file dynamically(because i dont know the number of files I will have at any point which is to be ftp'ed) and use it in Ftp command its not working properly.
---------- Post updated at 03:18 AM ---------- Previous update was at 03:14 AM ----------
Thanks Don.
Since I am polling a directory and don't have any idea of how may files will be there at any given moment, I will have to go by the approach of creating a dynamic file list and give it as input to the ftp command.
I try with your approach and get back here.
To answer your question., the file is already in EBCDIC format and just needs to be transferred in binary mode.
But there's old-fashioned ways to trick it into doing what you want: