App Deployment Setup on Ubuntu-Tomcat Cluster/Apache2 and MOD_JK Load Balance with Session Replication

This post can be supportive to setup a cluster of tomcat servers behind apache2 and load balanced using “mod_jk”, one of the apache modules, has been specifically designed to allow the forwarding of requests from Apache to Tomcat server or a servlet container, while maintaining sessions, which is most important for us here.

We are not deep diving into any of the setups of installing ubuntu and setting up FTP or SFTP. We understand that we have ubuntu server installed and putty is ready with a root login for setting up the deployment environment for web applications or .war file(s) run on a cluster of tomcat servers.

We can split the tasks as below,

1. Install JDK
2. Install Apache2
3. Install “mod_jk”
4. Install Tomcat Server
5. Set up Tomcat Cluster
6. Deploy sample web app
7. Load Balancer Configuration
8. Apache2 Configuration

Install JDK

If not already installed, let us install jdk on ubuntu to a default location, run the below command.

sudo apt-get install openjdk-8-jdk

Once installed, check the java version,

Install Apache2

Just start typing sudo apt-get install apache2, and leave all the default options. Once installation is completed, then check the configuration to see if everything installed right, use the command apachectl configtest …

You can ignore this message or if you want to fix then add ‘ServerName’ in the default config file of Apache2, in which we are going to do some changes down the line.

Install “mod_jk”

Do not wait any moment, just go ahead and hit the below command on the ubuntu terminal,

install libapache2-mod-jk

Once the installation is completed … … … wait a moment; we know that this component will be the key connector between apache and tomcat cluster, also acts as a load balancer. Now, we have Apache2 installed and mod_jk installed, we still need the tomcat cluster to be installed, why late? let us go there and back here again for configurations and setup.

Install Tomcat Server

Among multiple ways of installing Tomcat servers on ubuntu, we pick little straightforward for the use case. Let us download directly and unzip using below commands.

wget https://mirrors.estointernet.in/apache/tomcat/tomcat-8/v8.5.66/bin/apache-tomcat-8.5.66-windows-x64.zip

apt-get install unzip

unzip apache-tomcat-8.5.66-windows-x64.zip

Now, create a folder tomcluster or any name you wish, under /opt. Once created, copy the extracted tomcat folder apache-tomcat-8.5.66 to /opt/tomcluster.

Rename the folder apache-tomcat-8.5.66 to tomcat-1

Make a copy of tomcat-1 folder as tomcat-2.

So now we have tomcat-1 and tomcat-2 servers and we can cluster them.

Tomcat Cluster Setup

Let us first configure tomcat-1 server, the full path of tomcat-1 server would be /opt/tomcluser/tomcat-1.

Go to bin folder, create a file with name setenv.sh, add the below line and save it. Simply, we are setting JAVA_HOME here.

JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64

Once done, let us go to conf/server.xml file and modify configurations as needed for clustering of tomcat servers.

Set Shutdown port => 18005

<Server port=”18005" shutdown=”SHUTDOWN”>

Set up connection port => 18080

<Connector port=”18080" protocol=”HTTP/1.1" connectionTimeout=”20000" redirectPort=”18443" />

Set up AJP port => 18009

<Connector port=”18009" protocol=”AJP/1.3" redirectPort=”18443" secretRequired=”false”/>

Now, getting to actual cluster configuration section, the below highlighted is our point of interest, and here onwards we call ‘tomcat-1’ as jvm1.

<Engine name=”Catalina” defaultHost=”localhost” jvmRoute=”jvm1">

Just add the below Cluster configuration under the element <Host>.

<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">
    <Manager className="org.apache.catalina.ha.session.DeltaManager" expireSessionsOnShutdown="false"
             notifyListenersOnReplication="true"/>
    <Channel className="org.apache.catalina.tribes.group.GroupChannel">
        <Membership className="org.apache.catalina.tribes.membership.McastService" address="228.0.0.4"
                    port="45564" frequency="500" dropTime="3000"/>
        <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">
            <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"/>
        </Sender>
        <Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"
                  address="auto" port="4000" autoBind="100"
                  selectorTimeout="5000" maxThreads="6"/>
        <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"/>
        <Interceptor className="org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor"/>
        <Interceptor
                className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor"/>
    </Channel>
    <Valve className="org.apache.catalina.ha.tcp.ReplicationValve"/>
    <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"/>
    <Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
              tempDir="/opt/tomcluster/apps/temp"
              deployDir="/opt/tomcluster/tomcat-1/webapps"
              watchDir="/opt/tomcluster/apps"
              watchEnabled="true"/>
</Cluster>

Let us discuss, little bit on the “Deployer” section, from above configuration

<Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
              tempDir="/opt/tomcluster/apps/temp"
              deployDir="/opt/tomcluster/tomcat-1/webapps"
              watchDir="/opt/tomcluster/apps"
              watchEnabled="true"/>

Create apps folder under /opt/tomcluster, which will be used for deploying and un-deploying of applications in the cluster.

And, temp folder under /opt/tomcluster/apps, for tomcat server’s work, if any needed.

Set, watchEnabled=true on the first server or jvm1, we call as host.

NOTE=> all the other tomcat servers in the clusters like jvm2, jvm3 … has to use this flag as false.

With this, jvm1 or tomcat-1 or host server of the tomcat cluster configuration is predominantly completed.

Now, we can configure jvm2, the tomcat-2 server we created, similarly, with the ports 28005 (Shutdown), 28080 (Connect), 28443 (Redirect) and 28009 (AJP)

<Engine name=”Catalina” defaultHost=”localhost” jvmRoute=”jvm2">

Refer below Deployer configuration, for all node servers

<Deployer className="org.apache.catalina.ha.deploy.FarmWarDeployer"
              tempDir="/opt/tomcluster/apps/temp"
              deployDir="/opt/tomcluster/tomcat-2/webapps"
              watchDir="/opt/tomcluster/apps"
              watchEnabled="false"/>

NOTE => For all the node servers, watchEnabled=”false” and deployDir should be its own webapps deployment folder.

We configured, tomcat-1 (jvm1) as host and tomcat-2 (jvm2) as node servers of the Tomcat cluster.

Let us create a user and group on ubuntu and assign tomcat’s ownership. Refer below commands.

groupadd tomcat
useradd -s /sbin/nologin -g tomcat -d /opt/tomcluster tomcat
passwd tomcat
chown -R tomcat.tomcat /opt/tomcluster

Time to start the both jvms, navigate to /opt/tomcluster/tomcat-1/bin and /opt/tomcluster/tomcat-2/bin respectively to start the both servers.

sh startup.sh

Both the servers will be up and running on 18080 and 28080.

Sample web app Deployment

Take any sample webapp or .war file, and place it under /opt/tomcluster/apps.

The host server as listens on this folder, will pick up this .war file and deploy it over its webapps, and also deploys on all the nodes in the cluster.

The application is now deployed to all the tomcat server nodes in the cluster along with the host.

Now, we have app deployed to the servers, and apache webserver installed, also the apache2-tomcat connector too installed.

Load Balancer Configuration

Let us navigate to /etc/libapache2-mod-jk, and edit worker.properties. Copy the below properties as is and save the file. We are simply configuring two workers, jvm1 and jvm2 and pointing them to appropriate AJP ports we configured in the server.xml files of the tomcat servers accordingly.

Finally, we are coming to balance load among the workers, we have jvm1 and jvm2 for now. But any number of jvms can be added here, but they should be configured and up and running.

# First we define virtual worker’s list
worker.list=jkstatus, jvm1,jvm2, loadbalancer

# Enable virtual workers earlier
worker.jkstatus.type=status
worker.loadbalancer.type=lb

# Add Tomcat instances as workers, three workers in our case
worker.jvm1.type=ajp13
worker.jvm1.host=localhost
worker.jvm1.port=18009

worker.jvm2.type=ajp13
worker.jvm2.host=localhost
worker.jvm2.port=28009

# Provide workers list to the load balancer
worker.loadbalancer.balance_workers=jvm1,jvm2

Once saved the above, restart the apache2 server, by running command, systemctl restart apache2

Apache2 Configuration

Enable mod re-write for apache running the below command

a2enmod rewrite

Go to /etc/apache2/sites-enabled/ , open the file 000-default.conf

Add the below properties, under the <VirtualHost> section towards the end of the file

JkLogLevel debug

RewriteEngine on

RewriteRule ^/(.*)$ /$1 [PT,L]

JkMount /status status

JkMount /SampleWebApp loadbalancer

JkMount /SampleWebApp/* loadbalancer

NOTE=>

loadbalancer is a keyword used in the worker.properties file.

/SampleWebApp is the sample application deployed to tomcat cluster in the previous sections. (You can use any web application here, with proper access to resources)

The below screens are for demonstrative purpose and show how sessions get propagates in contrast with sticky session and tomcat cluster work.

With the above setup, we have configured a simple tomcat cluster with apache2 webserver load balanced by mod_jk.

Sticky sessions, which may cause uneven loads shared across clusters, now are well managed with the session propagation and above configuration/setup. Also the load is balanced among the two jvms. In case of failure of any of the servers, session will still intact and the other server shares the load as long as session is valid.

Many of the load balance properties can be explored referring to original apache sites, so as how to weight balance the load and other combinations to yield maximum performance.

Any time, the status of the cluster, if in flux, if the application might have removed manually or accidentally from one of the nodes or host, then make sure, un-deploy the application from the cluster by removing the .war file from the “apps” folder configured. And, copy the latest war file again to “apps” folder to hint the cluster to makes sure all nodes and host have the latest deployments and load balanced.

Thanks for reading the long post and I hope it may be helpful for the small scale setups and enthusiastic developers.

We will see in the next post, how we re-architecture the above design with the utilization of container capabilities and for high availability and scaling.