Howto: Clustering with JOnAS

Authors : Georges Goebel, Marc Calvisi, Rod Schultz, Jerome Pioux, Benoit Pelletier
October 16, 2006
Version : 3.2

Introduction

Architecture

        Used symbols 

        Load balancing at web level with mod_jk

        Session Replication at web level

        Load Balancing at EJB level

        Session Replication at EJB level

        Clustering databases with C-JDBC

SampleCluster2 Practical Example

        Products Installation

                Installing Java, Ant and JOnAS / Tomcat

                Installing Apache

               Installing the JK Module

        Configuring the network

        Configuring the Cluster

                Configuring the JOnAS nodes

                Configure the WebApp

                Database Concurrent Access

        SampleCluster2 Deployment

                Directory Structure

                Compiling sampleCluster2

        Running sampleCluster2

                Description of the sampleCluster2 Web Interface

References

This guide describes how to configure Apache, Tomcat, and JOnAS to install a cluster.

Clustering with JOnAS uses the following:

• the Apache/Tomcat plug-in mod_jk. This plug-in allows use of the Apache HTTP server in front of one or several Tomcat JSP/Servlet engines, and provides the capability of forwarding some of the HTTP requests (typically those concerning the dynamic pages, i.e., JSP and Servlet requests) to Tomcat instances.
  
  
• the In-Memory-Session-Replication technique based on the TCP based group communication protocol to provide failover at servlet/JSP level.
  
  
• the CMI protocol to support load balancing at EJB level for stateless beans and replications for stateful session beans.

This document describes one architecture with all the clustering functionalities available in JOnAS, the configuration of architectures integrating one of those functionalities, and other possible configurations.

For hands-on practice, this document uses the sampleCluster2 JOnAS example.

The architecture shown in the following illustration provides all the clustering functionality available in JOnAS: 1) Apache as the front-end HTTP server, 2) JOnAS/Tomcat as the J2EE Container, and 3) a shared database.

This architecture provides:

• Load balancing: Requests can be dispatched over a set of servers to distribute the load. This improves the "scalability" by allowing more requests to be processed concurrently.
  
  
• High Availability (HA): Having several servers able to fulfill a request makes it possible to ensure that, if a server dies, the request can be sent to an available server (thus the load-balancing algorithm ensures that the server to which the request will be sent is available). Therefore, "Service Availability" is achieved.
  
  
• Failover: This feature ensures that, at the web level (JSP/Servlet), if one server goes down another server is able to transparently take over.  Similarly, at the EJB level, if one server goes down another transparently takes over.  That is, requests will be switched to another server without service disruption, thus achieving Continuity.

 At the Servlet / JSP level, the mod_jk plug-in provides Load Balancing / High Availability and the Tomcat-Replication module provides Failover.
 At the EJB level, the CMI protocol and JNDI replication provides Load Balancing / High Availability.  Replication of Stateful EJBs provides failover.
 The database is shared by the JOnAS servers.

The sampleCluster2 application, presented later in this document, can be deployed on such an architecture.

The following illustration describes how Apache, mod_jk, and JOnAS/Tomcat Containers interact:

This example uses mod_jk, but an alternative configuration using Enhydra Director is also possible. (see config)

Mod_jk is a plug-in (module) that handles the communication and load balancing between Apache and Tomcat.

Mod_jk uses the concept of worker.  A worker is a Tomcat instance that is running to perform servlet requests coming from the web server.  Each worker is identified to the web server by the host on which it is located, the port where it listens, and the communication protocol that is used to exchange messages.

AJP13 is the preferred TCP/IP sockets protocol that mod_jk uses for communicating between the web server and Tomcat workers.

In this configuration there is one worker for each Tomcat instance and one worker (this is a specific worker with no host and no port number) that will handle the load balancing.  All workers are defined in a file called worker.properties.

Note: this module can also be used for site partitioning.

The following illustration shows that the HTTP session replication feature has been added to the architecture:

The term "session replication" is used when the current service state is being replicated across multiple application instances.  Session replication occurs when the information stored in an HttpSession is replicated from, in this example, one servlet engine instance to another.  This could be almost any type of data, such as items contained in a shopping cart or information being entered on an insurance application.  Anything being stored in the session must be replicated for the service to failover without a disruption.

The solution chosen for achieving Session replication is called all-to-all replication.  Tomcat uses a proprietary protocol based on TCP for the all-to-all replication.

The following illustration shows that load balancing at EJB level is now included in the architecture:

CMI is a new ORB used by JOnAS to provide clustering for load balancing and high availability.  It provides replication of the JNDI registries among all EJB containers.  Several instances of JOnAS can be started together in a cluster to share their EJBs.  It is possible to start the same EJB on each JOnAS, or to distribute their load.  A URL referencing several JOnAS instances can be provided to the clients.  At lookup time, a client randomly chooses one of the available servers to request the required bean.  Each JOnAS instance has the knowledge (through JGroups) of the distribution of the Beans in the cluster.  An answer to a lookup is a special clustered stub, containing stubs to each instance known in the cluster.  Each method call on the Home (or Remote for the SSB) of the bean can be issued by the stub to a new instance in order to balance the load on the cluster.  The default algorithm used for load distribution is currently a weighted round robin with local preference.

Session Replication at EJB level

The following illustration shows that the EJB session replication feature has been added to the architecture:

The term "session replication" is used when the current service state is being replicated across multiple application instances.  Session replication occurs when the information stored in an SFSB state is replicated from, in this example, one EJB container instance to another.  This could be almost any type of data (that is serializable), such as items contained in a shopping cart or information being entered on an insurance application.  Anything being stored in the session must be replicated for the service to failover without a disruption.

The solution chosen for achieving Session replication is based on a horizontal approach implemented both through CMI and the JOnAS's ha service. See here for more detailed information

With C-JDBC it is possible to cluster databases. Refer to HowTo setup a C-JDBC Database.

sampleCluster2 is an application that demonstrates the clustering features. A jsp/servlet follows the requests movements across the cluster. As shown in the illustration below, it can be deployed over four JOnAS instances.  Two have WEB containers and the other two have EJB containers.

The application is composed of three EJBs: a stateful session bean, a stateless session bean, and an entity bean.  These EJBs are instantiated by the web interface.  When the user connects to the web interface for the first time, an HTTP session is created.  During this process a stateful session bean is created in one of the JOnAS nodes where the EJB container is started.  The handle of this stateful session bean is added to the HTTP session (see note).  Next, the web interface creates a new instance of the stateless session bean.  The first time, no entity bean is created.  The useful information from the stateless session bean (JOnAS node name, number of instances) is set in the stateful session bean.

The user can see the result in the web interface where all the necessary information is displayed.  The user should now execute the same servlet several times in order to see the changes made by the cluster.  Each time the servlet is executed, it may be executed on a different node (JOnAS with web container).

The servlet gets a reference to the stateful session bean (through the handle). After that, a new stateless session bean is created and its information is passed to the stateful session bean.  On every tenth instantiation of the stateless session bean, an entity bean is created.

However,  the stateless session bean may be instantiated in a different EJB container than the stateful session bean.  This sample uses local interfaces between the stateless session bean and the entity bean.  As a result, the entity bean is created in the same EJB container as the stateless session bean.

The entity bean writes the time and the name of the node in the database.

Note:   The handle enables getting a reference to the stateful session bean when the JVM has changed.

See also description of the SampleCluster2 Web interface for a description of using sampleCluster2 to demonstrate cluster features.

This section provides information about installing JOnAS /Tomcat, Apache, mod_jk.

The versions assumed here are: JOnAS 4.8.x /Tomcat 5.5.x, Apache 2.2.x, and mod_jk 1.2.15 (or later).

1. Logon as any user <user> on the host <hostname> that will run the cluster.
   Note that you do not need to be ‘root’ to perform any of these steps or to run the cluster.
   
   
2. Enter  mkdir cluster
   to make a directory in which the supporting software (apache…) will be installed and the cluster will be deployed (in the next section also).
   
   
3. Enter  cd cluster
   to change directory to cluster.

Perform steps one through three in Installing JOnAS with a web container from scratch.

Linux Notes:

• For the sampleCluster2, JONAS_ROOT=$HOME/cluster/JONAS_4_8_3 (or later) is assumed.
  
  
• A JOnAS instance, also called a worker for mod_jk, is JONAS_ROOT and a JONAS_BASE.
  
  
• Even though there are four JOnAS instances (four JONAS_BASE), only one installation of JOnAS (one JONAS_ROOT) is needed.
  
  
• It is recommended that the environment variables set in the <user>’s .bash_profile (.profile on AIX) file be added.

export JAVA_HOME=<location where java was installed>

export ANT_HOME=<location where ant was installed>

export JONAS_ROOT=$HOME/cluster/JONAS_4_8_3

export PATH=$PATH:$JAVA_HOME/bin:$ANT_HOME/bin

Windows Notes:

• For the sampleCluster2,  JONAS_ROOT=c:\cluster\JONAS_4_8_3 (or later) is assumed.
  
  
• A JOnAS instance, also called a worker for mod_jk, is JONAS_ROOT and a JONAS_BASE.
  
  
• Even though there are four JOnAS instances (four JONAS_BASE) only one installation of JOnAS (one JONAS_ROOT) is needed.
  
  
• It is recommended that the environment variables set in the user environment variables
  My Computer | Properties | Advanced | Environment Variables be added.

JAVA_HOME   <location where java was installed>

ANT_HOME       <location where ant was installed>

JONAS_ROOT                   c:\cluster\JONAS_4_8_3

Add to PATH   ;$JAVA_HOME/bin;$ANT_HOME/bin

Installing Apache

For this sample, Apache will run as <user> (rather than 'root').
Below, <prefix> refers to the install folder for Apache.
This sample uses prefix = $APACHE_HOME= $HOME/cluster/apache2.

1. Download under cluster, the Apache server tarball source code from the Apache site.
   Extract the source code.
   tar xvfz httpd-2_0_XX.tar.gz
   
   
2. Choose an available listen port above 1024 (since this is not being run as ‘root’).
   This sample uses 8080. To verify that it is unused, use the following command.
   netstat -an | grep 8080
   
   
3. Configure, compile, and install.
   cd $HOME/cluster/httpd-2.2.0
   ./configure –-prefix=$HOME/cluster/apache2 --with-port=8080
   make
   make install
   
   Note: The $HOME/cluster/apache2/conf/httpd.conf file may need to be edited to change the "User" and "Group" lines to match the desired user and group.


Note: A binary version is also available at the Apache site.

Run and Stop Apache HTTP server to verify the installation.

1. Run Apache HTTP server.
   <prefix>/bin/apachectl start
   
   
2. Verify at least one httpd process is running.
   ps -ef | grep httpd
   
   
3. Verify that there is a listener.
   netstat -an | grep 8080
   
   
4. Use a web browser to connect to http://<hostname>:8080
   Check that apache returns the string ‘It Works!’
   
   
5. Stop Apache HTTP server for now – it will be restarted later after the JOnAS instances have been started.
   <prefix>/bin/apachectl stop

Notes:

• The following environment variables are not required but are convenient. They are used in sampleCluster2.
  export APACHE_HOME=$HOME/cluster/apache2
  export PATH=$PATH:$APACHE_HOME/bin
  
  
• It is also recommended that the <user>’s .bash_profile (.profile on AIX) file be updated with the lines above.

1. Download the Apache HTTP installer from the Apache site.
   The current version is apache_2.0.55-win32-x86-no_ssl.msi.
   
   
2. Choose a Listen port, for example, 8080
   
   
3. Enter  netstat -an  to verify it is unused. 
   If either port 80 or 8080 is not available, the <prefix>\Apache2\conf\http.conf file must be edited after installation. 
   Edit the line 'Listen 8080'.
   
   
4. Install by running the installer.
   The installer creates an 'Apache HTTP Server' group in the Start Menu.

Run and Stop Apache HTTP server:

1. Select  StartMenu -> Programs -> Apache HTTP Server -> Control Apache Server -- Start Apache Server in Console.
   
   
2. Open 'Task Manager', select 'Applications', verify 'Start Apache in Console' is running.
   
   
3. Enter netstat -an  to verify that there is a listener.
   
   
4. Stop Apache HTTP server.
   
   
5. Click on exit button on the windows title bar.
   
   

For further information about installing Apache HTTP server please refer to the Apache Site.

Mod_jk will be installed in the same <user> account as Apache.

1. Download under cluster, the mod_jk tarball source code from the Tomcat Site.  
   Extract the source code.
   tar xvfz jakarta-tomcat-connectors-1-2-15-src.tar.gz
   
   
2. Configure, compile and install.
   cd $HOME/cluster/jakarta-tomcat-connectors-1.2.15-src/native
   ./configure –-prefix=$APACHE_HOME --with-apxs=$APACHE_HOME/bin/apxs
   make
   make install

A mod_jk.so file should now have been created under $APACHE_HOME/modules.
The process for making apache aware of this new file will be performed in the next chapter.

Note: For some distributions, a binary version may be available at the Tomcat  site.

For further installation information, refer to the Tomcat Site.

Install mod_jk connector on Windows

1. Download the mod_jk connector.
   
   - Connect to the following url in a browser
        http://tomcat.apache.org/download-connectors.cgi
   - Select the browse download area link. This will take you to a mirror site.
   - Select jk
   - Select binaries
   - Select win32
   - Select  jk-1.2.15
   - Select mod_jk-apache-2.0.55.so to start the download.
   
   
2. Copy the downloaded file to <prefix>/apache2/modules.
   
   
3. Rename the file mod_jk.so.

For further installation information, refer to the Tomcat Site.

Configuring the Cluster

Configuring the JOnAS nodes

• Creating a JOnAS configuration manually
• Generating a JOnAS configuration with newjc

Configure the WebApp

Note: A WebApp deployed into a Cluster must have a special distributable element set in its WEB-INF/web.xml.
The following is an example of a web.xml deployment descriptor:

<web-app . . .>
  . . .
  <distributable />
  . . .
</web-app>

Please carefully read the Tomcat Clustering Guide for additional information.

Database Concurrent Access

Note: For sampleCluster2, in the JOnAS-specific deployment descriptor, tag shared is set to true for the entity beans involved.  When this flag is set to true, multiple instances of the same entity bean in different JOnAS servers can access a common database concurrently.

The following is an example of a deployment descriptor generated by newjc with the flag shared set to true. Deployment descriptors are in jonas_ejb_jar.xml in the ejbs.jar file in sampleCluster2.ear.

<jonas-ejb-jar>
  <jonas-entity>
           . . .

    <shared>true</shared>
           . . .

  </jonas-entity>
</jonas-ejb-jar>

sampleCluster2 is located under $JONAS_ROOT/examples/sampleCluster2.

/lib     libraries needed by the aplication to run

/output  generated ear, jar, war files

/src         source code

1. Logon as user <user>.
   Make sure this user has the Java, Ant, and JOnAS environment set (from the above installation).
   
   
2. cd $JONAS_ROOT/examples/sampleCluster2
   
   
3. export JONAS_BASE=$HOME/cluster/jb1
   to set JONAS_BASE for the first instance.
   
   
4. ant install
   
   As a result, the application file sampleCluster2.ear is copied to the $JONAS_BASE/apps/autoload directory for the first instance.
   Since the ear is copied to the autoload directory, deployment will be automatic at JOnAS startup.
   
   
5. Repeat steps 3 and 4 for the three remaining instances (steps will be shorter as the compile has been done in step 1).

Note: sampleCluster2 contains both a war file for the web application and a jar file for the EJB.  JOnAS will only deploy the war file to instances with a web container.  Similarly, it will only deploy the jar file on instances with an EJB container.

1. Logon as user <user>.
   
   
2. cd $JONAS_ROOT/examples/sampleCluster2
   
   
3. Start all four JOnAS instances:
   ./jcl4sc2 (jcl4sc2.bat for a Windows environment)
   
   This script executes
   jonas start –n nodeX, for X=1,2,3,4.
   
   
4. Restart Apache: <prefix>/bin/apachectl restart.
   
   
5. In a browser go to http://hostname:<apache listening port>/sampleCluster2

The following is an explanation of the information shown in the "SessionServlet Output" page above. 

Home of sampleCluster2

• Open a session:  creates a new session
  
• Release a session:  invalidates the http session and if a cookie is present, it will be made obsolete
  
• Check a session:  checks to see if a session is valid

After click on open session link

• getRequestURL:  indicates the number of requests for the indicated URL
  
• from <Client_IP>: Client requesting the page
  
• to <JOnAS_IP>:  JOnAS web-container that has executed the servlet
  
• Request path:
  -  Servlet: the node-name(-n nodeX) of the JOnAS web-container that has executed the servlet 
  -  Stateless EJB:  the node-name(-n nodeX) of the JOnAS ejb-container where the stateless session 
     bean has been created
  
• Session Data : New Session : false:  indicates if a new http session has been created in the web container
  
• Session ID:  id of the http session
  
• Creation Time:  date of the http session creation
  
• Last Accessed:  date of the last access to the session
  
• JOnAS instance=<node-name>:  ejb-container where the stateful ejb has been created
  
• Table
  -  servlet running on:  web-container that has executed the servlet
  -  stateless bean created:  ejb-container that has created the stateless session bean
  -  stateless bean total calls:  number of instantiations of the stateless bean on a specific node
  -  stateless bean instance count
  -  entity bean created:  if an entity bean has been created, this indicates the node-name of the EJB 
     container that created the entity bean links
  -  Ask again:  executes the servlet again
  -  release session:  invalidates the opened http session and if a cookie is present, it will be made obsolete
  -  check session:  checks to see if the session is valid
  -  home:  returns to the main page

After click on check session link

• JSessionID:  id of the http session
• links:
  -  go to session:  returns to the opened session
  -  check session:  checks to see if the session is valid
  -  home:  returns to the main page

• Working with mod_jk
• Tomcat workers Howto
• Apache JServ Protocol version 1.3 (ajp13)
• Apache - Tomcat HOWTO
• Apache 1.3.23 + Tomcat 4.0.2 + Load Balancing
• Tomcat 5.5 Clustering