URL call from SHELL script

Shell Programming and Scripting
1

My system does not have wget, curl and lynx. The chances are low that the SA would install them for me. Do I have any other way out? I need to invoke a few URL calls from my SHELL script.

Thought of using a PERL script or writing a simple Java client, but that adds in more dependencies.

Any idea?

2

if you have Python, you can use it's urllib2 library

#!/usr/bin/python
import urllib2
page = urllib2.urlopen("http://www.google.com")
result = page.read()
open("output.html","w").write(result) # write to output.html
3

Sorry, how do I know if I've python installed? Actually I just need to invoke an URL, thats all, no need to output it to a file.

4

you can type
# which python

or

# find / -name "python*".

5

For Perl, preferably LWP module is used for HTTP.

But for plain vanilla HTTP, if you can't really install LWP (note that you can download and install the module together with the script and not install it systemwide), a simple sequence of commands will do it:

use IO::Socket::INET; 
$http = IO::Socket::INET->new(
    PeerAddr => "www.unix.com:80",
    Proto => "tcp"
);
$http->print("GET /showthread.php?t=37182 HTTP/1.0\nHost: www.unix.com\n\n"); 
print $http->getlines();

which will display the HTML for this page. IO::Socket::INET is shipped standard with Perl, so you cannot say it is an "external dependency".

6

My URL call is actually invoking a servlet call. I don't really need to display any HTML. So does it mean that I should just omit this line of code?

print $http->getlines();
7

Unfortunately for me, Python is also not installed. Guess I'll have to use Perl then. Thanks!

8

a little crude

(sleep 4; echo "GET /path/to/html HTTP/1.1"; echo ""; sleep 5) | telnet webserver 80
9

I encountered the following message while running the perl script:

Any idea what does it means?

10

So if the URL call is:

http://abc-mmm.com/ctxroot/abc/servlet?task=xyz

Is this what I should be doing?

(sleep 4; echo "GET /ctxroot/abc/servlet?task=xyz HTTP/1.1"; echo ""; sleep 5) | telnet http://abc-mmm.com 80
11

remove "http://" from the telnet portion. only abc-mmm.com. the rest should be ok.

(sleep 4; echo "GET /ctxroot/abc/servlet?task=xyz HTTP/1.1"; echo ""; sleep 5) | telnet abc-mmm.com 80
12

I got this after trying:

What could be wrong?

13

Think the error was due to the server down. It's ok now. Thanks!

14

Your system does not have IO::Socket::INET installed. It is weird, because it is a standard module. So your system probably has an imcomplete, or very trimmed down installation of perl.

15

OP's version is very old. its version 5.005, and for this version, only Socket.pm is there. No INET.pm.

16

There was probably a restructuring of the modules since then ... I actually found IO::Socket::INET in the 5.5 documentation somewhere, but there are apparently no IO/Socket/INET.pm, so it seems the module exists, but require "use IO::Socket" instead of "use IO::Socket::INET". I no longer have this exact version to test the claim though.

Taken from 5.005 docs:
http://www.cse.msu.edu/doc/perl/lib/IO/Socket.html

17

How will i run the Python program ??

18

save the code as myscript.py, then on the command line,

 
# python myscript.py
19

You can install them in your home directory. Either download a binary (perhaps extract them from deb or rpm packages), or get the source code and compile them.

http://cfaj.freeshell.org/src/scripts/httpClient-sh

What do you mean by "invoke a few URL calls"? Do you mean, retrieve the files? View them in a rendered form?

20

Hi, I have to make call to servlet from either unix or perl,
my serlvet is returning me .zip file as part of the response.
can any body tell me how to capture the response and store it as a .zip file at the client machine.