It has been quiet on the outskirts lately, mainly due to excessively involved projects involving nothing but open source software.

A recent project involved integrating JBoss 7.0.2, Pentaho BI 4.0.0 GA (Tomcat), Apache, PHP 5, Perl 5.8, Oracle Forms 10g and Oracle Database 11g. This project involved load balancing utilizing both Apache Proxy as well as LVS and VMware ESXi. This was coupled with CUPS, Hylafax and some fun hardware configurations to make the whole thing as fault tolerant as possible.

Lessons learned in this project? CUPS wasn't meant to be clustered and forces you to be in a master-slave operation. LVS to multiple proxies (using WLC) adds an exciting new twist to debugging where your sessions are going. Sticky Sessions are easy on the application server, but a pain when running deployments and, everything gets more complicated in heterogeneous environments.

I guess if there were a problem to speak of, it would be about our IP addressing issue. We have a pair of Cisco ASA firewalls which provide IPSEC tunnels to various networks. Some of these networks we control, and some we don't. As there are a few hundred networks, it's excessively complicated to just ask them to change their settings. In fact, the last time we asked for a change - several sites took 6 months to get their changes in place. This really isn't acceptable and, we certainly can't afford to push this type of delay for the project release.

Working within the boundaries of the tunnel whereas each endpoint had access to only one IP address, we needed to solve the problem whereas the application server was printing directly to printers on the remote network. See, the old application allowed users to type in their printer name - like "abc003" and, cups would print to said printer. The printers were all networked and in the event that they were USB/Parallel or whatever, they were connected to their own print server somewhere.

The problem we ran into was that CUPS is just a queue. Sure you print to the queue, but a completely separate thread is responsible for connecting to the remote print server. Without modifying the "socket" source (or other cups backend applications), you're forced to use the default settings which are that the application connects to the remote endpoint via the "default" interface.

For reference, "default" applies to whatever interface the OS deems as appropriate for sending traffic to a remote endpoint. Routing 101 applies here....

Anyway, the result was that I created a failover cluster of print servers which is located behind a separate firewall. This firewall was running LVS and was responsible for forwarding traffic directly to the CUPS servers. The CUPS servers used a separate, private address in which they shared solely with the LVS firewall. When the firewall saw traffic to/from that private address range, then it used iptables to SNAT the traffic as the single IP that users connected to. As for the application? That was pushed behind the system too so that it could be on its own IP.

Then comes the new application, it's a bit more complicated - 3 BI servers, 3 application servers, 2 report servers, 2 database servers and 3 proxy servers. In this case, LVS on the aforementioned firewall just balances the port 80 traffic to one of the 3 proxy servers. Then, using sticky sessions, mod_proxy_balancer, mod_proxy_ajp and mod_headers to push the users to one of the application, report or bi servers (based on the URL) - I was able to enforce that one single IP address was seen by all endpoints both for inbound and outbound connections.

Oh, and for those who just say "well that's what NAT is for"... NAT was used only for the 3 print servers. "Direct Route" in LVS was used instead which puts a lower load on the load balancing servers and requires implementing arptables_jf on each of the servers involved...

Guess this is just a small excerpt of some of the stuff I do on a routine basis