Wget error while downloading from https website

Shell Programming and Scripting
1

Hi,

I would like to download a file from a https website. I don't have the file name as it changes every day.

I am using the following command:

wget --no-check-certificate -r  -np --user=ABC --password=DEF -O temp.txt https://<website/directory>

I am getting followin error in my temp.txt file:

I searched the entire forum and did google search but nothing helpful.

I plan to write shell script to automate this once I make this wget command to work.

Help is appreciated.

2

I suspect wget is working correctly and the site is malfunctioning.

It's difficult to say why without knowing what site, but I suspect --user and --password aren't doing what you want. They will not input passwords into form-elements.

3

Looks like website needs HTTPS POST Message.

I got following response from website administrator:

Help is appreciated with unix command to do this functionality.

I am using wget command not sure if this is the right command to use.

4

From man wget:

       --post-data=string
       --post-file=file
           Use POST as the method for all HTTP requests and send the specified
           data in the request body.  --post-data sends string as data,
           whereas --post-file sends the contents of file.  Other than that,
           they work in exactly the same way. In particular, they both expect
           content of the form "key1=value1&key2=value2", with percent-encod-
           ing for special characters; the only difference is that one expects
           its content as a command-line paramter and the other accepts its
           content from a file.

The real question is, what data does this website expect in POST? There is no generic answer, that is up to their implementation, you'll have to look at its <form>s to see that.

5

Looks like for the call to website I have to encode my password as follows:

first convert to uppercase,
then Unicode it in little-endian UTF 16,
Then SHA1 it then base64 encode it.

Unicode encoding = new UnicodeEncoding();
hashBytes = encoding.GetBytes(password.ToUpper().Trim());
SHA1 sha1 = new SHA1CryptoServiceProvider();
byte[] cryptPassword = sha1.ComputeHash(hashBytes);
String pwd= Convert.ToBase64String(cryptPassword);

Can we do all this in unix or will have to use some other technology. If all this can be done in unix.

Help is appreciated.

6

Wow, they're really taking the long, long, long way around. Though I can kind of send the point of sending the hash, not the password -- except if anyone steals that hash instead, that's just as useful. In other words everyone that looks at the page without a browser sees right through it...

7 
STR=$(  printf asdf                     |
        tr 'a-z' 'A-Z'                  |
        iconv -f "UTF-8" -t "UTF-16LE"  |
        sha1sum                         |
        awk '{ for(N=1; N<=length($1); N+=2) printf("\\x" substr($1,N,2)); }'
        )

BASE64=$(printf "$STR" | openssl base64)

echo $BASE64
8

Thanks, I put the commands you provided in a shell and executed it.

I got the following error messages.

Looks like sha1sum is not installed on our server.
Is it a free software or a licensed one?

iconv is installed on our server not sure, why I am getting error for "cannot open converter".

Appreciate help.

9

sha1sum is open-source and freely available. I thought it was part of openssl, but apparently it's in GNU Coreutils. It may also be available as part of busybox (which many Linux systems keep around for emergencies).

Try iconv -l to see what codes it can convert between.

10

You could also use openssl sha1 :

printf asdf                             |
        tr 'a-z' 'A-Z'                  | 
        iconv -f "UTF-8" -t "UTF-16LE"  |
        openssl sha1 -binary            |
        openssl base64