Recently I was trying to pull data from a website. As usual, the website had a HTML page with input text boxes and after providing the necessary input, it shows up the response.

Looks simple! Right? Well, It wasn’t. Actually I wanted to pull numbers of records ( 100+ ). I realized it does not make sense to enter inputs every time and wait for the server response. I wanted to do it quick!

Looking into the page source, I found that it was indeed a simple HTML form with POST method. The basic structure of HTML form was

<form action=”xyz.asp” method=”POST”>

<input name=’param1’ value=’’/>
<input name=’param2’, value=’’/>

</form

In short, my requirement was to write a program which should take values for parameters ‘param1’ and ‘param2’ and hits the action page using POST method!

All I wanted was a simple program which should do the job for me ( See, how lazy I am

After going through a bit of perl and java documentation, I decided to write the script in perl because of it’s simplicity ( yet powerful ). Here’s the script if you have the similar problem.

1 #!/usr/bin/perl
2 use strict;
3 use warnings;
4 use LWP::UserAgent;
5 use HTTP::Request::Common qw(POST);

6 my $ua = LWP::UserAgent->new;
7 push @{ $ua->requests_redirectable }, ‘POST’;
8 my $request = POST ‘http://www.aaaa.com/xyz.asp’, [ param1 => 'abcd', param2 => '10341'];
9 my $response = $ua->request($request);

10 if ($response->is_success) {
11 print $response->content;
12 }
13 else {
14 print $response->status_line . “\n”;
15 }
16 exit;

Few things to note:

1) If you want the method as GET, just replace the POST with GET in script.
2) Line 7 is required only if the website does the redirection for results. It’s always safe to include cause you may never know whether the website which you are going to hit does redirection for response.
3) The script assumes that the website does not require authentication.
4) Script uses LWP::UserAgent class. Hence, you will need LWP package for perl.
- For ActiveState Perl users, you will have to fire up your Perl Package Manager and install the package.
- For unix/linux users, you can directly connect to CPAN and install the package. The following command can help
$ perl –MCPAN –e shell
$ install LWP::UserAgent

BTW, I wrote the same thing in Java. I’ll be posting it later

Tags: Java, Perl, Programming, Technology, Web