Hi,
My script needs to crawl the data from a third party site. Currently it is written in wget. The third party site is of shared interface with different IP addresses.
My wget works with all the IP address but not with one. Whereas the curl is able to hit that IP address and comes out with 200 status.
Hence I would like to know why my wget is not working whereas the curl succeeds for the same IP address.
How can i resolve this Problem, i don't want my script to get rewritten with curl. Your help at this moment is so huge, thanks in advance.
$ wget --spider https://XXX.YYY.com
Spider mode enabled. Check if remote file exists.
--2012-12-12 04:51:48-- https://XXX.YYY.com/
Resolving XXX.YYY.com... 23.62.197.60
Connecting to XXX.YYY.com|23.62.197.60|:443...
It hungs and comes out reaching the timeout.
PS: I have added the Proxy in my ~/.wgetrc file
$ cat ~/.wgetrc
http_proxy=http://str-www-proxy2:8080
curl -v -x http://str-www-proxy2:8080 https://XXX.YYY.com
* About to connect() to proxy str-www-proxy2 port 8080
* Trying 165.130.174.102... connected
* Connected to str-www-proxy2 (165.130.174.102) port 8080
* Establish HTTP proxy tunnel to XXX.YYY.com:443
> CONNECT XXX.YYY.com:443 HTTP/1.0
> Host: XXX.YYY.com:443
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 Connection established
---------- Post updated at 09:14 AM ---------- Previous update was at 04:57 AM ----------
Any help please ??