Assistance with Perl and HTTP

Shell Programming and Scripting
1

I need to query a http site and then parse the xml results, this works well if I use the string in IE but I require an automated solution.

I have tried using the following as well as HTTP::Request, nothing seems to work any suggestions would be appreciated, I have tried diffrnt things I found on the site but it seems as though I can't connect even though I can from IE.
<begin>
use LWP::Simple;
$content = getprint("http://www.web.com/feeds/val.php?vid=val&pw=val&op=val&reporttype=val&qp=val");
die "Couldn't get it!" unless defined $content;
<End>

2 
tyler@mecgentoo ~ $ perldoc -f getprint
No documentation for perl function `getprint' found
tyler@mecgentoo ~ $

I love the perl help system.

Have you tried downloading the page with wget? The page might block anything that's not IE or Firefox.

3

What exactly is not working in your case? Did you examine the error code? You can also put in tcpdump to examine the http request-response pair to check what is happening.

Corona688,

The getprint() is actually a static method of LWP::Simple that is exported to the function namespace using the Exporter module. Therefore, you don't need to qualify with the module name. It is not an intrinsic function but can be called like a function if LWP::Simple is loaded.

4

By the way, according to the LWP::Simple manpage, the return value of getprint() is the HTTP status code. The HTTP response is sent to STDOUT. If you want to get the response in a string, you have multiple methods:

  1. Use get() instead of getprint().
  2. Use the full-featued LWP instead of LWP::Simple.
  3. Perform output buffering and capture the STDOUT in a scalar variable.

e.g.

my $content;
{
    local *STDOUT;
    open(STDOUT, '>', \$content);
    $status = getprint("http://www.google.com");
}
# $status contains HTTP status code
# $content contains HTTP response
5

I am getting a 500 Can't connect to www.google.com:80 (Connect: Unknown error) < URL:Http://www.google.com>

6

Are you on a peculiar Windows security configuration set by your network administrator that only IE works? It all works fine for me. Or check whether you have a firewall e.g. ZoneAlarm that blocks all network connections except those initiated by applications you have previously agreed to.

If everything fails, check the packets by tcpdump then. I am inclined to think it is not a problem of LWP::Simple or Perl.

7

The only thing I can think of is IE is aware of our proxy and the perl isn't, Can you specify a proxy in Perl?

8

The LWP::UserAgent module has a proxy attribute for specifying HTTP and FTP proxy.

This also means you cannot use LWP::Simple. If so, you will need LWP.