Information Technology Management & Development

A Simple Perl Proxy Server

Praveen Ray — Tue, 27 Jul 2010 20:04:04 +0000

A simple perl proxy server which listens for web requests and log them to a file. Very useful for debugging web requests on the server side. Especially traditional form POSTs , which are not displayed by FireBug.

Usage:

> ./proxy-server.pl

Change your browser to use a proxy server. Point it to the host running this script and port 8001.

#!/usr/bin/perl
 
use strict;
use HTTP::Proxy qw(:log);
use HTTP::Proxy::BodyFilter::simple;
use HTTP::Proxy::BodyFilter::complete;
use Data::Dumper;
 
my $proxy = HTTP::Proxy->new(port => 8001);
$proxy->host(undef);
#$proxy->logmask(ALL);
 
$proxy->push_filter(
    mime => undef,
    response => HTTP::Proxy::BodyFilter::complete->new
);
 
$proxy->push_filter(
    mime => undef,
    request => HTTP::Proxy::BodyFilter::simple->new(sub {
        my ($self, $dataref, $req, $protocol, $buffer ) = @_;
        open (my $fh, '>>', ($ARGV[0] || 'http-recorder.txt'));
        my $cookie = $req->header('cookie');
 
        my $xhr    = $req->header('x-requested-with');
        print $fh $req->method,' ',$req->uri;
        my @headers;
 
        if($cookie) {
            push(@headers, ['cookie', $cookie]);
        } 
        if ($xhr) {
            push(@headers, ['x-requested-with', $xhr]);
        }
	my $content_type = $req->header('content-type');
	if($content_type) {
		push(@headers, ['content-type', $content_type]);
	}
        if(@headers > 0) {
            local $Data::Dumper::Indent=0; 
            local $Data::Dumper::Varname=""; 
            my $str = Data::Dumper::Dumper(\@headers);
            $str =~ s/^\$1\s+=\s+//;
            print $fh " HEADERS: $str" if $str;
        }
        print $fh 'CONTENT: ',$req->content if $req->content;
        print $fh "\n";
        $fh->close;
    }),
	response => HTTP::Proxy::BodyFilter::simple->new(sub {
       my ($self, $dataref, $req, $protocol, $buffer ) = @_;
	   open (my $fh, '>>', 'http-response.txt');
	   print $fh $$dataref, "\n";
	   $fh->close;
    })
);
 
# start the proxy
$proxy->start();
 
1;

It listens on port 8001 and writes headers and incoming data (including POSTed data) in a file called http-recorder.txt and outgoing response to http-response.txt

GWT Development with Traditional Servlets

Praveen Ray — Fri, 25 Jun 2010 14:04:11 +0000

If you’re developing GWT apps with traditional servlets (other than GWT-RPC) with your web server running as separate process, you can’t really send GWT Ajax messages from hosted mode since the outside process and port are considered to be another domain and GWT won’t send cross domain requests. One clean solution is to use a transparent proxy as hosted mode servlet and let that proxy talk to the real server. Here are the steps I took:
Find following jars are copy them to your GWT project:

{GWT-Project}/war/WEB-INF/lib
        jetty-http-7.1.3.v20100526.jar
        jetty-io-7.1.3.v20100526.jar
        jetty-servlets-7.1.3.v20100526.jar
        jetty-continuation-7.1.3.v20100526.jar
        jetty-util-7.1.3.v20100526.jar
        jetty-client-7.1.3.v20100526.jar
        jetty-jmx-7.1.3.v20100526.jar

Add these jars to Referenced Libraries -> Build Path -> Configure Build Path -> Libraries -> Add Jars.
Don’t forget to make sure these jars are selected in ‘Order and Export’ Tab.

Modify {GWT-Project}/war/WEB-INF/web.xml:

>
	>
	    >JettyContinuationFilter>
	    >org.eclipse.jetty.continuation.ContinuationFilter>
	>
	>
	    >JettyContinuationFilter>
	    >/app/*>
	>
    >
        >jetty-proxy-servlet>
        >org.eclipse.jetty.servlets.ProxyServlet$Transparent>
        >
            >ProxyTo>
            >http://localhost:8080/>
        >
        >
            >Prefix>
            >/>
        >
        >1>
    >
>
        >jetty-proxy-servlet>
        >/app/*>
    >   
>

Now all your URLS in your GWT app starting with ‘/app’ are normally forwarded to your traditional web server at localhost:8080.
BTW, I got these jetty jars from my local maven .m2 repository.

GWT Server Servlet ClassNotFoundException

Praveen Ray — Fri, 04 Jun 2010 20:01:56 +0000

If you’re getting ClassNotFoundException for your GWT server servlets, even though the class exists and is ok, you might want to check out the Build Path in Eclipse and make sure All libraries are selected in

 Java Build Path -> Order and Export.

The error message could be confusing and misleading. Hope this tip saves you some time.

Spring Unit Tests

Praveen Ray — Wed, 26 May 2010 14:22:39 +0000

Dependency Injection is supposed to make your life easier when it comes to writing Unit and Integration Tests; assuming you do your homework right and indeed make use of Injection. With Spring 3 annotations like @Autowired and configurations like , there really is no excuse for sloppiness.
Here’s is how to reap the rewards of DI when writing unit tests. You can dependency inject each Test Class using same @Autowired annotation that you use for your normal classes.

import org.junit.runner.RunWith;
import org.springframework.test.context.ContextConfiguration;
 
@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration({"classpath:spring-config.xml"})
public class MyClassTest {
   @Autowired
   private TransferMoneyStrategy strategy = null;
   ....
   @Test
   public void test_transfer() {
          // write Asserts here
   }
}

Key is the annotation @ContextConfiguration – one line is all you need to boot up entire Spring infrastructure!

HSQLDB woes

Praveen Ray — Wed, 05 May 2010 02:07:18 +0000

Is it just me or HSQLDB really has issues with BLOBs in general. I tried using it with Jetty and the server would throw random IndexOutOfBound Exceptions. Not the client but HSQLDB server! It would become unresponsive after the exception and nothing short of dropping and recreating the database would work.

Quite disappointing I must say. I tried running it with a connection pool of one but still same issue.

Going back to MySQL . It’s a shame since HSQL is so much easier to bundle and distribute.

CSV Parsing with Erlang

Praveen Ray — Sat, 07 Nov 2009 14:00:47 +0000

Since OTP doesn't provide any CSV parsing capabilities, I decided to write my own based upon gen_fsm. Following the short explanation given in Perl's Text::CSV_XS module, I implemented a simple state machine. Erlang's excellent binary processing capabilities and built in gen_fsm behavior make the code compact, and surprisingly easy to implement. You can download the code from here. A short explanation follows.

The state machine has only following handful of states:

start_field

Start reading a CSV field – it might be double quoted and might have special chars such as \r, \n, comma and double quote. Goto read_field or read_quoted_field, depending upon if a double quote started this field.

read_field

Once a field has started, we switch to this state and read binary bytes until and end of field condition is detected. End of Field is marked by either a comma or a newline.

read_quoted_field

We're inside a double quoted field; read everything until another double quote is encountered. A double quote might be end of this field or an embedded double quote marked with two consecutive double quotes. Switch to escaped_double_quote if a double quote is encountered.

escaped_double_quote

We come inside this field upon encountering a double quote inside read_quoted_field state. If another double quote is seen, it's an escaped double quote, else, it's nothing special. Both these cases go back to read_quoted_field.

Usage

Following public methods are exported:

parse_csv(File_path)
parse_csv(Binary_blob)
parse_csv(File_path, Options)
parse_csv(Binary_blob, Options)

where Options is:

[{callback_fn, Fun}, {callback_state, term()}]

Fun is a function/2 and gets called with a List of Fields and callback state.

With no Options passed, the return is a list of list of Fields. With callback_fn passed, the Callback is called at the end of each line with a list of Fields.

Examples:

parse_csv:parse_csv("/tmp/data.csv").

Returns:

[[<<"Date">>,<<"Source">>,<<"Destination">>,
  <<"Seconds">>,<<"CallerID">>,<<"Disposition">>,<<"Cost">>],
 [<<"2009-09-18 09:44:54">>,<<"5097213333">>,
  <<"18667778888">>,<<"66">>,<<"5098761323">>,<<"ANSWERED">>,
  <<"0">>]]

F = fun(Fields, State) -> io:format("~p~n",[Fields]), State + 1 end.

parse_csv:parse_csv("/tmp/data.csv", [{callback_fn, F},{callback_state, 0}]).

It calls F repeatedly. First with second parameter set to 0, then 1, then2 and so on. Note that your fun must return a modified state which becomes second parameter to callback_fn for the next line.

Send Emails without installing SMTP server

Praveen Ray — Thu, 29 Oct 2009 17:57:52 +0000

Sending emails is easy. All you need is access to an SMTP server. Or local Sendmail daemon. Installing and configuring either of the two is a massive task. Even if you get it right, your ISP might not like you running SMTP server. If you are past ISP hurdle, chances are your emails will get marked as SPAM for lack of Reverse DNS.

This is an easier way and we all know it – just use Google SMTP server. All you need is a gmail account or Google Apps account – and who doesn’t have a google account?

However, sometimes, the third party software you’re trying to configure to send emails doesn’t support anything other than localhost. Or it does but wants plain SMTP whereas Google does only TLS. Or simply, connecting to Google’s SMTP for each email takes few seconds and you want faster response.

Here’s a simple solution:

Install the wonderful Email Relay program from here

Create a file :

vi /etc/google.email.auth

and add following line:

login client yourgoogleemailid@gmail.com yourgooglepassword

chown daemon:root /etc/google.email.auth
chmod 400 /etc/google.email.auth

run emailrelay as:

emailrelay --as-proxy smtp.gmail.com:587 --client-tls --client-auth /etc/google.email.auth

Configure your software to use localhost for SMTP server (Port defaulted to 25).

Test your Local Emailer. If it doesn’t work, enable logging in /etc/emailrelay.conf by uncommenting verbose line and look into your syslog file (/var/log/syslog)

Moving your wordpress blogs

Praveen Ray — Wed, 28 Oct 2009 21:58:16 +0000

Recently I had to create a clone of this blog site so we can apply upgrades and test these out without fear of breaking our actual blogs. Here’s a list of steps I followed. Hopefully, it’ll help others.

Assumptions:

existing blog URL: http://a.com.
New blog URL: http://b.com/wordpress
Existing blog is installed in /var/www
New blog will be installed in /var/www/wordpress
The MySQL database for wordpress content is called ‘wordpress’. Username and passwords are also ‘wordpress’

ssh root@a.com
 cd /var/
 tar cf wordpress.tar www
 bzip2 wordpress.tar
 mysqldump --add-drop-table-uwordpress -pwordpress -Dwordpress -hlocalhost >; db.backup
 bzip2 db.backup
 scp wordpress.tar.bz2 db.backup.bz2 root@b.com:/var/www
ssh root@b.com
 cd /var
 tar jxf wordpress.tar.bz2
 mysql -uroot -p<mysql root="" pw="">         
 create database wordpress;
 GRANT ALL PRIVILEGES ON wordpress.* to 'wordpress'@'localhost' identified by 'wordpress*'; 
 exit;  
 cd /var/www  bunzip2 db.backup.bz2 
 mysql -uwordpress -p'wordpress' -Dwordpress -hlocalhost < db.backup  
 cd /var/www
 mkdir wordpress 
 cd wordpress 
 mv ../wordpress.tar.bz2 .   
 tar jxf wordpress.tar.bz2  
 mv www/* .  
 rm -rfwordpress.tar.bz2 www     
 cd /var/www    
 mysql-uwordpress -p'wordpress' -Dwordpress -hlocalhost
    update wp_posts set guid=REPLACE(guid,'http://a.com','http://b.com/wordpress');         
    update wp_options set option_value='http://b.com/wordpress/' where option_name='siteurl';   
    exit;

You should be able to login as admin at http://b.com/wordpress/wp-admin

Goto Settings/General and change all instances of http://a.com to http://b.com/wordpress

You might also have to goto your currently selected theme and make sure URLs are changed. Some themes have URLs embedded in them.

A Simple REST controller for Spring MVC

Praveen Ray — Wed, 14 Oct 2009 18:23:22 +0000

RESTful programming is all the rage these days – at least in Rails world. It's a simple way to perform request dispatching which is easy to understand and maps database CRUD operations into web requests elegantly. While Rails comes with REST routing built in, there is no such support in Spring MVC. However, Spring's decoupled design makes it rather easy to implement. Here's code I wrote to implement RESTful dispatches using Spring MVC.(Also available as a download here)

package com.yellowfish.servlets;
import javax.servlet.http.*;
 
import org.apache.log4j.*;
 
import java.util.*;
 
import org.springframework.web.servlet.mvc.multiaction.*;
 
public class RESTMethodResolver implements MethodNameResolver {
 
    public String getHandlerMethodName(HttpServletRequest req) throws NoSuchRequestHandlingMethodException {
 
        Logger log = Logger.getLogger("com.yellowfish.servlets");
 
        log.debug("Resolving method name for the Request");
 
        String method = req.getParameter("_method");
 
        String id     = req.getParameter("id");
 
        String http_method = req.getMethod().toUpperCase();
 
        String controller_method = null;
 
        log.debug("method: "+method+" http_method: " + http_method + " id: "+id);
 
        if(http_method.equals("GET")) {
 
            String uri = req.getRequestURI();
 
            controller_method =  (uri.endsWith("/new") ? "_new" : "show");
 
            if(controller_method == "show") {
 
                controller_method = (id == null) ? "index" : "show";
 
            }
 
        } else if(http_method.equals("POST")) {
 
            if(method != null) {
 
                method = method.toUpperCase();
 
                if(id == null)
 
                    throw new com.yellowfish.servlets.ServletException("id cannot be NULL for PUT and DELETE");
 
                if (method.equals("PUT"))
 
                    controller_method = "update";
 
                else if(method.equals("DELETE"))
 
                    controller_method = "delete";
 
            } else {
 
                controller_method = "create";
 
            }
 
        }
 
        if(controller_method == null) {
 
            log.debug("No Controller Method Found");
 
            throw new NoSuchRequestHandlingMethodException(null, this.getClass());
 
        }
 
        log.debug("Controller method resolved to :" + controller_method);
 
        req.setAttribute("controller_method_name", controller_method);
 
        if(id != null) {
 
            try {
 
                req.setAttribute("model-id", Long.valueOf(id));
 
            } catch(NumberFormatException exp) {
 
                log.warn("id" + id + " is not a Number");
 
            }
 
        }
 
        return controller_method;
 
    }
 
}

Then simply declare this bean in the spring config:

<bean class="com.yellowfish.servlets.RESTMethodResolver" id="rest-method-resolver">bean>

It resolves incoming URL and Method as per following table:

REST Resolver Rules
URI	GET/POST	ID Parameter Present?	_method Parameter Present?	Method Name
/server	GET	No	N/A	index
/server/_new	GET	No	N/A	new
/server?id=100	GET	Yes	N/A	show
/server	POST	No	No	create
/server	POST	Yes	PUT	update
/server	POST	Yes	DELETE	delete

To make life simple, the code above assumes all IDs to be numeric and if ID is present in incoming request, it creates a Request attribute called 'model-id' .

Chicanery on Wall St

Praveen Ray — Mon, 12 Oct 2009 16:51:49 +0000

Reading the book ‘Traders Guns & Money‘ by Satyajit Das. It’s an awesome account of greed and ‘anything goes’ attitude on Wall St (and financial markets around the globe). From ultra complex to borderline illegal instruments concocted by Dealers and Investment bankers around the world to make quick killings , everything that unraveled three years after the book was written, is detailed by the author. It’s interesting, he wrote about these years before the meltdown. But so did Warren Buffet who called these derivatives the financial WMDs!

Some of my favorite quotes from the book:

The most intriguing thing about convertible bonds is that everybody seems to be getting a great deal. It’s a win-win. Markets don’t work that way. Question is, who is fooling whom?

Robert Citron, treasurer of Orange County before it went bust, confident Interest rates wouldn’t rise.: ‘I am one of the largest investors in America, I know these things’

Same guy testifying during investigations, after Orange county lost 1.5B: ‘My brain was unable to audit’ !

Traders work on the theory that what is mine is mine, and what is yours is also mine.

The irony of trying to model chaos, the finding of order in complete disorder, is lost on most quants.