Automated scp using shell & expect

veeresh_15 · October 27, 2016, 9:58am

Hi All,

I have written a script to automate scp of files. Most of the times it works fine except few cases. I want your help and suggestions to fix these failures.

I have used expect & shell to do the automated scp. Below is code


$ cat scp.ksh
#!/bin/ksh
inputfile=$1
fdest_sid=$2
echo -n "Enter Fully Qualified Source Server Name:"
read -r rserver
echo -n "Enter userid of source server:"
read -r ruser
echo -n "Enter password for source server:"
stty -echo
read -r rpassword
stty echo
echo  ""
export rserver
export ruser
export rpassword

export logfile=file_transfer_$(date +%Y%M%d_%H%M%S).log
> $logfile

cnt=$(cat $inputfile|wc -l)
var=$(cat $inputfile|tr '\n' ' '|sed -e 's/.$//g')
scp_cmd=$(echo "scp $ruser@$rserver:'${var}' $fdest_sid")
./expect_scp "$scp_cmd" "$cnt" >> $logfile 2>&1




$ cat expect_scp
#!/usr/bin/expect
set scp_cmd [lindex $argv 0]
set cnt [lindex $argv 1]
set rpass "$env(rpassword)\r"
set prompt "(%|#|\$) $"
set j 0
spawn ksh
send "$scp_cmd\r"
expect "Are you sure you want to continue connecting (yes/no)?" {send "yes\r"}
expect "*assword:"  {send "$rpass\r"}
while {$j < $cnt} {
        expect {
                "ETA" {
                        puts "still transferring..."
                        exp_continue;        
                        }
                "100%" {
                        puts "done"
                        }
                }
        incr j
}
expect "*$"
send {"exit\r"}

In file transfer log, i can see the shell command that i pass to expect script are send to the spawned shell with some control characters.

eg:

command passed to expect script:

scp oracle@server.domain.com:'/FS1/DB1/ETL_STAGING_A01.dbf /FS1/DB1/KALIDO_IN_D03.dbf /FS2/DB1/ETL_STAGING_A02.dbf /FS2/DB1/KALIDO_IN_D04.dbf /FS1/DB1/ETL_STAGING_A03.dbf /FS1/DB1/KALIDO_IN_D05.dbf /FS2/DB1/ETL_STAGING_A04.dbf /FS2/DB1/KALIDO_IN_D06.dbf /FS1/DB1/ETL_STAGING_A05.dbf' /FS3/DB2

Expect Script sending below command to shell:

$ scp oracle@server.domain.com:'/FS1/DB1/ETL ^H_STAGING ^H_A01.dbf /FS1/DB1/KALIDO ^H_IN ^H_D03.dbf /dbbcnprod/or^M$ oradata02/DB1/ETL ^H_STAGING ^H_A02.dbf /FS2/DB1/KALIDO ^H_IN ^H_D04.d                                                                                 <^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^Hbf /FS1/DB1/ETL ^H_STAGING ^H_A03.dbf /FS1/DB1/                                                                                ^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M$ WH/KALIDO ^H_IN ^H_D05.dbf /FS2/DB1/ETL ^H_STAGING ^H_A04.dbf /dbbcnprod/o                                                                                ^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^Hradata02/DB1/KALIDO ^H_IN ^H_D06.dbf /FS1/DB1/ETL ^H_STAGING ^H_A05.^M$  ^H_D06.dbf /FS1/DB1/ETL ^H_STAGING ^H_A05.dbf' /u02/oradata03/DB2                                                                                ^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^H^M

These control charcters are causing scp failures some times.
How do i get rid of these control characters?

Regards,
Veeresham

rbatte1 · October 27, 2016, 10:21am

First of all, and without reading the rather wide code/output, I would respectfully suggest that you should not try to force your way through scp with expect. Tools built like scp, ssh, sftp etc. have security designed in to them and you are trying to break/bypass them in an insecure manner.

You would be far better generating keys to automate the process. It will also make your code far more sensible. Is there a reason not to use the tools they way they are designed? If your script is interactive, why not get the tools to do the work for you?

Robin

RudiC · October 27, 2016, 10:57am

I'd rather check the input file if it contains unusual characters that, via tr , sed , $var and an unnecessary "command substitution", end up in the $scp_cmd .

And, I second rbatte1, in that that is an overly complicated approach. Did you, on top of the suggestions he made, consider using sftp for multi file transfer?

veeresh_15 · October 27, 2016, 11:05am

Thanks Robin & Rudi for your comments. I will think of using key generation technique to do this. But still want to know from where these junk characters are coming.

Rudi,

Here is the input file. I generate this file by querying a DB table.

/FS1/DB1/ETL_STAGING_A01.dbf
/FS1/DB1/KALIDO_IN_D03.dbf 
/FS2/DB1/ETL_STAGING_A02.dbf 
/FS2/DB1/KALIDO_IN_D04.dbf 
/FS1/DB1/ETL_STAGING_A03.dbf 
/FS1/DB1/KALIDO_IN_D05.dbf 
/FS2/DB1/ETL_STAGING_A04.dbf 
/FS2/DB1/KALIDO_IN_D06.dbf 
/FS1/DB1/ETL_STAGING_A05.dbf

Regards,
Veeresham

Corona688 · October 27, 2016, 11:15am

If you install keys, the procedure for copying without a password is:

scp localfile username@hostname:destinationfile

...that's it.

It's that simple, because that's the way it's intended to be used -- not forcing it with a third-party brute force hacking tool.

RudiC · October 27, 2016, 11:19am

Unfortunately, control chars are not that easily seen in test posts. Show us the result of

od -ctx1 $inputfile

and/or

echo $var | od -ctx1

.

veeresh_15 · October 27, 2016, 3:57pm

Hi Rudi,

Below is the output of od command

$ od -ctx1 2.dfscp_saa
0000000   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d   a
         2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64  61
0000020   t   a   0   2   /   P   R   O   D   D   W   H   /   E   T   L
         74  61  30  32  2f  50  52  4f  44  44  57  48  2f  45  54  4c
0000040   _   S   T   A   G   I   N   G   _   A   0   6   .   d   b   f
         5f  53  54  41  47  49  4e  47  5f  41  30  36  2e  64  62  66
0000060  \n   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d
         0a  2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64
0000100   a   t   a   0   2   /   P   R   O   D   D   W   H   /   K   A
         61  74  61  30  32  2f  50  52  4f  44  44  57  48  2f  4b  41
0000120   L   I   D   O   _   I   N   _   D   0   8   .   d   b   f  \n
         4c  49  44  4f  5f  49  4e  5f  44  30  38  2e  64  62  66  0a
0000140   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d   a
         2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64  61
0000160   t   a   0   1   /   P   R   O   D   D   W   H   /   E   T   L
         74  61  30  31  2f  50  52  4f  44  44  57  48  2f  45  54  4c
0000200   _   S   T   A   G   I   N   G   _   A   0   7   .   d   b   f
         5f  53  54  41  47  49  4e  47  5f  41  30  37  2e  64  62  66
0000220  \n   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d
         0a  2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64
0000240   a   t   a   0   1   /   P   R   O   D   D   W   H   /   K   A
         61  74  61  30  31  2f  50  52  4f  44  44  57  48  2f  4b  41
0000260   L   I   D   O   _   I   N   _   D   0   9   .   d   b   f  \n
         4c  49  44  4f  5f  49  4e  5f  44  30  39  2e  64  62  66  0a
0000300   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d   a
         2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64  61
0000320   t   a   0   2   /   P   R   O   D   D   W   H   /   E   T   L
         74  61  30  32  2f  50  52  4f  44  44  57  48  2f  45  54  4c
0000340   _   S   T   A   G   I   N   G   _   A   0   8   .   d   b   f
         5f  53  54  41  47  49  4e  47  5f  41  30  38  2e  64  62  66
0000360  \n   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d
         0a  2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64
0000400   a   t   a   0   2   /   P   R   O   D   D   W   H   /   K   A
         61  74  61  30  32  2f  50  52  4f  44  44  57  48  2f  4b  41
0000420   L   I   D   O   _   I   N   _   D   1   0   .   d   b   f  \n
         4c  49  44  4f  5f  49  4e  5f  44  31  30  2e  64  62  66  0a
0000440   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d   a
         2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64  61
0000460   t   a   0   1   /   P   R   O   D   D   W   H   /   E   T   L
         74  61  30  31  2f  50  52  4f  44  44  57  48  2f  45  54  4c
0000500   _   S   T   A   G   I   N   G   _   A   0   9   .   d   b   f
         5f  53  54  41  47  49  4e  47  5f  41  30  39  2e  64  62  66
0000520  \n   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d
         0a  2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64
0000540   a   t   a   0   1   /   P   R   O   D   D   W   H   /   K   A
         61  74  61  30  31  2f  50  52  4f  44  44  57  48  2f  4b  41
0000560   L   I   D   O   _   I   N   _   D   1   1   .   d   b   f  \n
         4c  49  44  4f  5f  49  4e  5f  44  31  31  2e  64  62  66  0a
0000600   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d   a
         2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64  61
0000620   t   a   0   2   /   P   R   O   D   D   W   H   /   E   T   L
         74  61  30  32  2f  50  52  4f  44  44  57  48  2f  45  54  4c
0000640   _   S   T   A   G   I   N   G   _   A   1   0   .   d   b   f
         5f  53  54  41  47  49  4e  47  5f  41  31  30  2e  64  62  66
0000660  \n   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d
         0a  2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64
0000700   a   t   a   0   2   /   P   R   O   D   D   W   H   /   K   A
         61  74  61  30  32  2f  50  52  4f  44  44  57  48  2f  4b  41
0000720   L   I   D   O   _   I   N   _   D   1   2   .   d   b   f  \n
         4c  49  44  4f  5f  49  4e  5f  44  31  32  2e  64  62  66  0a
0000740   /   d   b   b   c   n   p   r   o   d   /   o   r   a   d   a
         2f  64  62  62  63  6e  70  72  6f  64  2f  6f  72  61  64  61
0000760   t   a   0   1   /   P   R   O   D   D   W   H   /   E   T   L
         74  61  30  31  2f  50  52  4f  44  44  57  48  2f  45  54  4c
0001000   _   S   T   A   G   I   N   G   _   A   1   1   .   d   b   f
         5f  53  54  41  47  49  4e  47  5f  41  31  31  2e  64  62  66
0001020  \n
         0a
0001021

$ cat 2.dfscp_saa
/dbbcnprod/oradata02/PRODDWH/ETL_STAGING_A06.dbf
/dbbcnprod/oradata02/PRODDWH/KALIDO_IN_D08.dbf
/dbbcnprod/oradata01/PRODDWH/ETL_STAGING_A07.dbf
/dbbcnprod/oradata01/PRODDWH/KALIDO_IN_D09.dbf
/dbbcnprod/oradata02/PRODDWH/ETL_STAGING_A08.dbf
/dbbcnprod/oradata02/PRODDWH/KALIDO_IN_D10.dbf
/dbbcnprod/oradata01/PRODDWH/ETL_STAGING_A09.dbf
/dbbcnprod/oradata01/PRODDWH/KALIDO_IN_D11.dbf
/dbbcnprod/oradata02/PRODDWH/ETL_STAGING_A10.dbf
/dbbcnprod/oradata02/PRODDWH/KALIDO_IN_D12.dbf
/dbbcnprod/oradata01/PRODDWH/ETL_STAGING_A11.dbf

RudiC · October 28, 2016, 6:02am

OK, there don't seem to be any odd characters. What about the $var contents?

rbatte1 · October 28, 2016, 6:31am

Are you trying to copy the database files to clone the database? This could be achieved but it's not as simple as a straight copy. Assuming that this is an Oracle database you need to do a few things before you copy to ensure that the files are idle and you have to consider the target server too has to recognise the files in the correct location, along with any parameter file adjustments that might be needed.

Can I ask what the overall aim of your project is? There may be better ways or specific steps you need to follow.

Apologies if you already have it working and this is just an attempt to smooth the process.

Robin