Background (skip this if you just want to get to the server setup details)
My group does custom web applications for use the Operating Rooms. We interface with multiple clinical and administrative systems and show, collect, and share data that's critical to the operations and to patient safety. Our longest running application, ORview, is a Java web app that collects pre-operative assessments and post-operative assessments, provides airport monitor-style big screen views throughout the OR area, and provides other billing, reporting and QA/QI functionality. Our newest application is RequestOR, which collects posting requests for new case postings, and shares, via HL7, this data with GE's Centricity Periopertive Manager and IDX. Our old ORview prod server was at end-of-life, and we were needing a new server to deploy RequestOR on so we designed, spec'd and installed a new server cluster that'll meet the needs of both of our major applications, and give us some room to migrate some of our more minor applications (Java and JRuby).
Server Hardware
Qty 2 - IBM HS21 Blade Server, Dual Xenon Quad-core @ 3.0GHz, 8GB PC2-5300 RAM, 146G SAS HDD.
Server Software
Red Hat Enterprise Linux Server release 5.3 (Tikanga)
Java SE Runtime Environment 64-bit (build 1.6.0_13-b03)
Apache HTTPd 2.2.3 (httpd-2.2.3-22.el5_3.1) with mod_ssl (mod_ssl-2.2.3-22.el5_3.1) and mod_proxy_ajp (as part of the httpd install)
Apache Tomcat 6.0.18
Network/DNS Configuration
The key to doing multiple SSL-secured applications is that each unique SSL certificate NEEDS it's own IP address and host name to bind to. (There are some budding ways around that but none of those were mature enough going into this process to be a viable production option.)
requestor: 10.20.215.226, 10.20.215.228 - configured as dns round-robin
orview2: 10.20.215.227, 10.20.215.229 – configured as dns round-robin
Those IPs get distrubuted to each server that's hosting that application, along with each server having their own ip.
server 1
10.20.215.223 – blade1
10.20.215.226 – requestor
10.20.215.227 – orview2
228.0.0.23 - multicast
server 2
10.20.215.224 – blade2
10.20.215.228 – requestor (dns rr)
10.20.215.229 – orview2 (dns rr)
228.0.0.23 - multicast
Apache HTTPd Configuration
First, the explanation. There's a default port 80 host (in black) that just answers requests on the machine's unique IP. We use this index.html to point to a simple machine ident. The RequestOR section is next (in blue). It defines a port 80 host that redirects all requests to the port 443 (ssl) version of itself. The second virtual host in the blue is the ssl version. This section contains config info for the SSL certificates, and a reference to balancer://ajpCluster/requestor as handler for all requests to this virtualhost. The ORview2 section (in red) largely duplicates this configuration for the second ssl application on these servers. The second server's config files are identical, except the IPs are changed to match that server's configuration.
<VirtualHost *:80>
DocumentRoot /var/www/html
ServerName blade1.foo.edu
</VirtualHost>
# RequestOR virtual hosts
<VirtualHost 10.20.215.226:80>
DocumentRoot /var/www/html/requestor
ServerName requestor.foo.edu
Redirect permanent / https://requestor.foo.edu/
</VirtualHost>
<VirtualHost 10.20.215.226:443>
DocumentRoot /var/www/html/requestor
ServerName requestor.foo.edu
SSLCertificateFile
/etc/pki/tls/certs/requestor.foo.edu.crt
SSLCertificateKeyFile
/etc/pki/tls/private/requestor.foo.edu.key
SSLEngine on
SSLProtocol all -SSLv2
<Location />
ProxyPass balancer://ajpCluster/requestor
stickysession=JSESSIONID
</Location>
# ErrorLog logs/dummy-host.example.com-error_log
# CustomLog logs/dummy-host.example.com-access_log common
</VirtualHost>
#ORview2 virtual hosts
<VirtualHost 10.20.215.227:80>
DocumentRoot /var/www/html
ServerName orview2.foo.edu
Redirect permanent / https://orview2.foo.edu/
</VirtualHost>
<VirtualHost 10.20.215.227:443>
DocumentRoot /var/www/html
ServerName orview2.foo.edu
SSLCertificateFile /etc/pki/tls/certs/orview2.foo.edu.crt
SSLCertificateKeyFile
/etc/pki/tls/private/orview2.foo.edu.key
SSLEngine on
SSLProtocol all -SSLv2
<Location />
ProxyPass balancer://ajpCluster/orstat stickysession=JSESSIONID
</Location>
</VirtualHost>
DocumentRoot /var/www/html
ServerName orview2.foo.edu
Redirect permanent / https://orview2.foo.edu/
</VirtualHost>
<VirtualHost 10.20.215.227:443>
DocumentRoot /var/www/html
ServerName orview2.foo.edu
SSLCertificateFile /etc/pki/tls/certs/orview2.foo.edu.crt
SSLCertificateKeyFile
/etc/pki/tls/private/orview2.foo.edu.key
SSLEngine on
SSLProtocol all -SSLv2
<Location />
ProxyPass balancer://ajpCluster/orstat stickysession=JSESSIONID
</Location>
</VirtualHost>
Apache mod_proxy_ajp Configuration
This is the config for the ajp load balancer. Tomcat on each server is set to listen for AJP requests on port 8009. This config file (the same on each server) tells the AJP balancer about the cluseter composed of TomcatA and TomcatB. In the absence of any other details, it'll default to sending new requests to the least loaded Tomcat server, and sending requests from existing sessions to the server that's been handling them. That's what the stickysessions attribute makes happen. Proxy listeners are configured for each application, same color scheme as above.
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
#
# When loaded, the mod_proxy_ajp module adds support for
# proxying to an AJP/1.3 backend server (such as Tomcat).
# To proxy to an AJP backend, use the "ajp://" URI scheme;
# Tomcat is configured to listen on port 8009 for AJP requests
# by default.
#
<Location /balancer-manager>
SetHandler balancer-manager
</Location>
<Proxy balancer://ajpCluster>
BalancerMember ajp://blade1.foo.edu:8009 route=tomcatA
BalancerMember ajp://blade2.foo.edu:8009 route=tomcatB
</Proxy>
<Location /requestor>
ProxyPass balancer://ajpCluster/requestor stickysession=JSESSIONID
</Location>
<Location /orstat>
ProxyPass balancer://ajpCluster/orstat stickysession=JSESSIONID
</Location>
#
# When loaded, the mod_proxy_ajp module adds support for
# proxying to an AJP/1.3 backend server (such as Tomcat).
# To proxy to an AJP backend, use the "ajp://" URI scheme;
# Tomcat is configured to listen on port 8009 for AJP requests
# by default.
#
<Location /balancer-manager>
SetHandler balancer-manager
</Location>
<Proxy balancer://ajpCluster>
BalancerMember ajp://blade1.foo.edu:8009 route=tomcatA
BalancerMember ajp://blade2.foo.edu:8009 route=tomcatB
</Proxy>
<Location /requestor>
ProxyPass balancer://ajpCluster/requestor stickysession=JSESSIONID
</Location>
<Location /orstat>
ProxyPass balancer://ajpCluster/orstat stickysession=JSESSIONID
</Location>
Tomcat's server.xml
The server.xml is generally a good sized file where most of the defaults are just fine. I've excerpted the relevant bits here that I had to change to get clustering working.
A connector is defined in the <server> portion of the file. It should be enabled by default.
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />
There should be one <engine> element. Edit it to include the unique name used for this in the proxy-ajp config file - tomcatA in this case. The other server's server.xml looks the same except for tomcatB here.
<Engine name="Catalina" defaultHost="localhost" jvmRoute="tomcatA">
A cluster element is defined inside the <engine> element. The important thing to set here is the multicast address (in red) to be used by Tribes to synchronize session information across the servers in the cluster. The FarmWarDeployer (in blue) is experimental and (as of when this was written) isn't ready for prime-time.
<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"
channelSendOptions="6">
<Manager className="org.apache.catalina.ha.session.BackupManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"
mapSendOptions="6"/>
<!--
<Manager className="org.apache.catalina.ha.session.DeltaManager"
expireSessionsOnShutdown="false"
notifyListenersOnReplication="true"/>
-->
<Channel className="org.apache.catalina.tribes.group.GroupChannel">
<Membership className="org.apache.catalina.tribes.membership.McastService"
address="228.0.0.23"
port="45564"
frequency="500"
dropTime="3000"/>
<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
address="auto"
port="5000"
selectorTimeout="100"
maxThreads="6"/>
<Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
<Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
</Sender>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"/>
<Interceptor className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/>
</Channel>
<Valve className="org.apache.catalina.ha.tcp.ReplicationValve"
filter=".*\.gif;.*\.js;.*\.jpg;.*\.png;.*\.htm;.*\.html;.*\.css;.*\.txt;"/>
<Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
tempDir="/tmp/war-temp/"
deployDir="/tmp/war-deploy/"
watchDir="/tmp/war-listen/"
watchEnabled="false"/>
<ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
</Cluster>
Conclusion
It's completely possible to get high-availibility load balancing and clustering working for Apache's HTTPd and Tomcat under Linux. The performance and fault-tolerant benefits are completely worth it. and thanks to a lot of work done by a lot of dedicated people, it's pretty easy to get set up and running. I send my thanks to every site I Googled figuring out how to get this working - there are too many to count. If you have questions or corrections, please post them here, and I'll do what I can to help figure them out.
