Hi,
There is a LB on AWS. There are 2 hosts behind this LB, and URL is https://sc-ecrf.int.dev-multi.state911.net
When apps guys are running "wget <value for lat/long in xml file>" from Solaris 10 (x86), they are seeing below error
#Remote Query
rlost_url=https://sc-ecrf.int.dev-multi.state911.net
echo "Sending a Remote HTTP POST to the xLoST server URL:"
echo -e "$rlost_url"
sleep 1
rstartd=`date +'%H:%M:%S'`
/usr/local/bin/wget --post-file=/export/home/prodadm/xytools/$1 $rlost_url -O /export/home/prodadm/xytools/remote_lost.ashx >> $rlogfile 2>&1
rendd=`date +'%H:%M:%S'`
I am not able to figure if this is something on Solaris server (being using old TLS), or something should be checked on AWS side.
Can someone give some suggestion?
Thanks
s_client
This implements a generic SSL/TLS client which can establish a transparent connection to a remote server speaking SSL/TLS . It's intended for testing purposes only and provides only rudimentary interface functionality but internally uses mostly all functionality of the OpenSSL ssl library.
from man openssl
NB: I have no Solaris + limited in depth messing around with OpenSSL.
I think myself there are two things that are the most likely here: either the SSL/TLS protocol supported by the server is not one supported by the client; or the server is in fact serving unencrypted HTTP on port 443, and not encrypted HTTPS.
From the client if you telnet to port 443 on that server and issue a regular GET / HTTP/1.0 command, do you get a response, or do you get disconnected ? If you get a response this would mean that you are actually getting HTTP serviced on what is expected to be an HTTPS port.
I am able to connect this way, if you meant this -
bash-3.2# telnet sc-ecrf.int.dev-multi.state911.net 443
Trying 10.134.128.103...
Connected to sc-ecrf.int.dev-multi.state911.net.
Escape character is '^]'.
^]
telnet> q
Connection to sc-ecrf.int.dev-multi.state911.net closed.
bash-3.2#
To check, if SSL/TLS protocol supported by the server or not, I checked this link -
bash-3.2# openssl s_client -help 2>&1 > /dev/null | egrep "\-(ssl|tls)[^a-z]"
-ssl3 - just use SSLv3
-tls1_2 - just use TLSv1.2
-tls1_1 - just use TLSv1.1
-tls1 - just use TLSv1
bash-3.2#
But I didn't get clearly, if it is giving some clue.
and pressing Return a few times. If you get back human-readable text (most likely an HTTP header at the minimum), then the server is serving HTTP on port 443, which is almost certainly the problem.
If you just get disconnected after issuing the GET command then it probably is serving HTTPS, and an unsupported protocol then becomes the most likely explanation.
I hit enter twice, and it was disconnected and came back on prompt
bash-3.2# telnet sc-ecrf.int.dev-multi.state911.net 443
Trying 10.134.128.103...
Connected to sc-ecrf.int.dev-multi.state911.net.
Escape character is '^]'.
GET / HTTP/1.0
HTTP/1.1 400 Bad Request
Server: awselb/2.0
Date: Fri, 20 Oct 2023 00:15:45 GMT
Content-Type: text/html
Content-Length: 220
Connection: close
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
</body>
</html>
Connection to sc-ecrf.int.dev-multi.state911.net closed by foreign host.
bash-3.2#
Ah, there we are. From your response, it looks like the server is indeed serving HTTP (what you normally get served on port 80) on port 443 (which would normally be serving encrypted traffic via HTTPS).
So to be clear: SSL/TLS/etc. is not being used at all on that load-balanced virtual IP on port 443. It's just serving plain clear-text HTTP on 443. If it were configured to respond to HTTPS only on 443, you wouldn't have gotten back a human-readable response to your HTTP GET request.
This is almost certainly a misconfiguration that needs fixing at the server side, or the load balancer side. If I were to take a guess as to what's gone wrong here, I'd say that the load balancer is indeed talking to servers at the back-end that are serving HTTPS on port 443, but the VIP that the load balancers are providing is configured to serve HTTP on port 443 of the VIP and not HTTPS.
Anyway - to immediately make the error disappear on the Solaris clients, you can use a URL like http://sc-ecrf.int.dev-multi.state911.net:443/ . This URL will tell a Web client to use HTTP on port 443, which is apparently what the load balancer is configured to serve on this VIP.
In terms of a proper long-term fix, the load balancer and/or back-end server config needs updated to either actually not bother serving anything on port 443 at all if SSL isn't in use, or the load balancer needs to serve actual encrypted traffic on port 443 and not just respond in plain-text HTTP, as this will confuse many/most Web clients to no end.
Thank you so much. https://sc-ecrf.int.dev-multi.state911.net and servers behind this LB, are managed by different team. They pushed it to me saying I should be checking something on Solaris server, or its cert or TLS support or something else. But as mentioned, it seems like I need to put it back to them, with suggestions you made.
Nice to see teamwork does lead to solving issues...
Welcome solaris_1977 to the hard work of system team, often having to show their issue is not theirs...
I checked with apps guys and who maintains LB and backend server. He replied - The load balancer is configured for TLS (as the message you got back indicated - you sent a request to an HTTPS port but didn't use HTTPS).
My best guess for the cause of the issue is that wget is not supporting TLS 1.2, so it's failing to negotiate TLS w/ the load balancer. Is there anyway you can update it or otherwise enable TLS 1.2?
Yes, I am reading it now.
I am reading a note saying -
If the user "wants to use TLS1.2" then he needs to check where he wants to use this. The OpenSSL package is just a library and will not change any used application, i.e. if the user wants or needs TLS 1.2 then he needs to reconfigure his used application.
So I tried to run wget manually. It looks like it accepts "TLSv1", but not "TLSv1_2"
bash-3.2# openssl s_client -help 2>&1 > /dev/null | egrep "\-(ssl|tls)[^a-z]"
-ssl3 - just use SSLv3
-tls1_2 - just use TLSv1.2
-tls1_1 - just use TLSv1.1
-tls1 - just use TLSv1
bash-3.2#
bash-3.2# /usr/local/bin/wget --secure-protocol=TLSv1
wget: missing URL
Usage: wget [OPTION]... [URL]...
Try `wget --help' for more options.
bash-3.2#
bash-3.2# /usr/local/bin/wget --secure-protocol=TLSv1_2
wget: --secure-protocol: Invalid value `TLSv1_2'.
bash-3.2# /usr/local/bin/wget --secure-protocol=TLSv1.2
wget: --secure-protocol: Invalid value `TLSv1.2'.
bash-3.2#
Support for TLS 1.1 and 1.2 was added to OpenSSL 1.0.1 and I have 1.02 -
bash-3.2# openssl version
OpenSSL 1.0.2r 26 Feb 2019
bash-3.2#
Hmm, that's interesting. Perhaps the load balancer software has intelligence that allows it to discern between an HTTP and an HTTPS request on the same port, and it then acts accordingly. I've not encountered that personally in the past on Linux-based load balancers, but it could well be that whatever appliance this is does have such intelligence built-in.
In terms of default behaviours, what I've always seen in the past when trying to send clear text to an encrypted socket is basically this:
# telnet localhost 443
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.0
Connection closed by foreign host.
#
As can be seen, the connection is simply closed after the clear-text request is sent, because what the server is expecting to receive is an SSL/TLS handshake, and not a clear-text HTTP request. Any time in the past I've gotten back a genuine HTTP header and/or body in response to a manual HTTP request over telnet, it has always meant that the server is communicating in the clear and not over SSL. But as I say, perhaps they are right, and their load balancer has the intelligence to send an appropriate response on any socket.
I am not sure, if apps guy can use curl in their code, need to check with them. But curl is not installed on this servers.
For wget, here is what I see for version and ldd
bash-3.2# /usr/local/bin/wget --version
GNU Wget 1.12 built on solaris2.10.
+digest +ipv6 +nls +ntlm +opie +md5/solaris +https -gnutls +openssl
+iri
Wgetrc:
/usr/local/etc/wgetrc (system)
Locale: /usr/local/share/locale
Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
-DLOCALEDIR="/usr/local/share/locale" -I. -I../lib
-I/usr/local/include -I/usr/local/ssl/include
-I/usr/local/include/ncurses -I/usr/openwin/include
-I/usr/local/rrdtool-1.2.19/include
-I/usr/local/BerkeleyDB.4.7/include -I/usr/local/include/lzo -O2
-L/usr/local/lib -R/usr/local/lib -L/usr/local/ssl/lib
-R/usr/local/ssl/lib -L/usr/openwin/lib -R/usr/openwin/lib
-I/usr/local/rrdtool-1.2.19/include
-I/usr/local/BerkeleyDB.4.7/include
Link: gcc -O2 -L/usr/local/lib -R/usr/local/lib -L/usr/local/ssl/lib
-R/usr/local/ssl/lib -L/usr/openwin/lib -R/usr/openwin/lib
-I/usr/local/rrdtool-1.2.19/include
-I/usr/local/BerkeleyDB.4.7/include -L/usr/local/lib
-R/usr/local/lib -R/usr/lib -L/usr/lib -R/usr/openwin/lib
-L/usr/openwin/lib -L/usr/local/ssl/lib -R/usr/local/ssl/lib
-L/usr/X11R6/lib -R/usr/X11R6/lib -L/usr/local/BerkeleyDB.4.7/lib
-R/usr/local/BerkeleyDB.4.7/lib /usr/local/ssl/lib/libssl.so
/usr/local/ssl/lib/libcrypto.so -R/usr/local/ssl/lib
/usr/local/lib/libiconv.so -L/usr/local/BerkeleyDB.4.2/lib
-R/usr/local/lib /usr/local/lib/libintl.so
/usr/local/lib/libiconv.so -L/usr/local/BerkeleyDB.4.2/lib
/usr/lib/libsec.so /usr/lib/libc.so -R/usr/local/lib -R/usr/lib
-R/usr/openwin/lib -R/usr/local/ssl/lib -R/usr/X11R6/lib -lmd5 -ldl
-lsocket -lnsl -lrt -lidn ftp-opie.o openssl.o http-ntlm.o
gen-md5.o ../lib/libgnu.a
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Currently maintained by Micah Cowan <micah@cowan.name>.
Please send bug reports and questions to <bug-wget@gnu.org>.
bash-3.2#
bash-3.2# ldd /usr/local/bin/wget
libssl.so.0.9.8 => /usr/local/ssl/lib/libssl.so.0.9.8
libcrypto.so.0.9.8 => /usr/local/ssl/lib/libcrypto.so.0.9.8
libiconv.so.2 => /usr/local/lib/libiconv.so.2
libintl.so.8 => /usr/local/lib/libintl.so.8
libsec.so.1 => /usr/lib/libsec.so.1
libc.so.1 => /usr/lib/libc.so.1
libmd5.so.1 => /usr/lib/libmd5.so.1
libdl.so.1 => /usr/lib/libdl.so.1
libsocket.so.1 => /usr/lib/libsocket.so.1
libnsl.so.1 => /usr/lib/libnsl.so.1
librt.so.1 => /usr/lib/librt.so.1
libidn.so.11 => /usr/lib/libidn.so.11
libgcc_s.so.1 => /usr/local/lib/libgcc_s.so.1
libavl.so.1 => /lib/libavl.so.1
libmp.so.2 => /lib/libmp.so.2
libmd.so.1 => /lib/libmd.so.1
libscf.so.1 => /lib/libscf.so.1
libaio.so.1 => /lib/libaio.so.1
libdoor.so.1 => /lib/libdoor.so.1
libuutil.so.1 => /lib/libuutil.so.1
libgen.so.1 => /lib/libgen.so.1
libm.so.2 => /lib/libm.so.2
bash-3.2#
bash-3.2# pkginfo -l | grep -i wget
PKGINST: SMCwget
NAME: wget
bash-3.2# pkginfo -l SMCwget
PKGINST: SMCwget
NAME: wget
CATEGORY: application
ARCH: x86
VERSION: 1.12
BASEDIR: /usr/local
VENDOR: Free Software Foundation
PSTAMP: Steve Christensen
INSTDATE: Sep 24 2018 15:07
EMAIL: steve@smc.vnet.net
STATUS: completely installed
FILES: 150 installed pathnames
6 shared pathnames
86 directories
2 executables
5090 blocks used (approx)
bash-3.2#
In comparison, if I check same on Linux 7.9, it accepts TLS 1.2. But wget is 1.14 there.
[root@rhel_7.9 ~]# /usr/bin/wget --version
GNU Wget 1.14 built on linux-gnu.
+digest +https +ipv6 +iri +large-file +nls +ntlm +opie +ssl/openssl
Wgetrc:
/etc/wgetrc (system)
Locale: /usr/share/locale
Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
-DLOCALEDIR="/usr/share/locale" -I. -I../lib -I../lib -O2 -g -pipe
-Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
--param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic
Link: gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector-strong --param=ssp-buffer-size=4
-grecord-gcc-switches -m64 -mtune=generic -lssl -lcrypto
/usr/lib64/libssl.so /usr/lib64/libcrypto.so /usr/lib64/libz.so
-ldl -lz -lz -lidn -luuid -lpcre ftp-opie.o openssl.o http-ntlm.o
../lib/libgnu.a
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Please send bug reports and questions to <bug-wget@gnu.org>.
[root@rhel_7.9 ~]# wget --secure-protocol=TLSv1_2
wget: missing URL
Usage: wget [OPTION]... [URL]...
Try `wget --help' for more options.
[root@rhel_7.9 ~]#