CMI (Clustered Method Invocation)

Introduction

CMI is the protocol cluster for JOnAS ensuring:

• the jndi high availability through the registry replication and the multi-target lookup
• the load-balancing and fail-over at the EJB level (RMI flow) through the CMI cluster stub
  • for the Home interface of the EJB's SSB, SFSB, EB
  • for the Remote interface of the EJB's SSB
• the SFSB high availability with the JOnAS HA service

Getting started with the CMI Protocol

CMI can be enabled in JOnAS by:

• setting the cmi protocol in the $JONAS_BASE/conf/carol.properties file
• compiling an application with the CMI protocol

Clustered objects and CMI Registry

CMI brings its own registry for implementing the jndi replication. Each CMI registry instance over the cluster contains two areas:

• a local area for hosting the local objects that must not be replicated
• a distributed area hosting the global objects (cluster objects) that must be replicated

When an object is registered in the registry, the routing to the local or global area is done according to:

• the type of the object: only the remote objects are replicated
• the presence of the 'CMI class' having the same name as the object class with the suffix '_CMI' and inherited from the org.objectweb.carol.cmi.Distributor class
• one of the two following methods returning true:
  • equivAtBind() if the replication has to take place at the bind() time. For example, this could be the case for the EJB Remote Home objects.
  • equivAtExport() if the replication has to take place at the export() time. Typically, this could be the case for the SSB Remote objects.

The entries of the distributed area are lists providing the ability, for example, to gather several stubs for the the same jndi-name and thus to return a stubs list.

JNDI HA

Registry Replication

CMI relies on JGroups group-communication protocol for ensuring the global registry replication. The parameters are gathered in the:

• $JONAS_BASE/conf/carol.properties for specifying the JGroups configuration file name and the JGroups group name.
• $JONAS_BASE/conf/jgroups-cmi.xml file for the settings of the jgroups protocol stack. By default, the JGroups configuration uses the UDP protocol and the multicast IP for broadcasting the registry updates. A TCP-based stack can be used in a network environment that does not allow the use of multicast IP or when a cluster is distributed over a WAN.

All the members of a cluster share the same JGroups configuration.

If several cluster partitions are required over a single LAN, several JGroups configurations must be configured with different values for the following parameters:

• JGroups group name
• JGroups multicast address
• JGroups multicast port

When a new node appears in the cluster, its registry content is synchronized automatically.

When a node disappears, JGroups notifies the other's member of the node leaving and the registry entries related to this node are removed.

Registry Fail-over

On the client side, the high availability of the registry is provided by the capability to set several JOnAS instances in the registry url. At the lookup time, the client chooses (round-robin algorithm) one of the available servers to get the home stub. If the server fails, the request is sent to another server. The CMI url registry is specified in the $JONAS_BASE/conf/carol.properties file using the following syntax:

carol.cmi.url=cmi://server1:port1[,server2:port2...] 

CMI Cluster Stub or Cluster-aware Stub

Load-balancing and fail-over on the client side are provided through cluster-aware stubs. These stubs are generated on the fly through ASM and rely on:

• the RMI/JRMP protocol for communicating with the server side,
• the CMI class associated with the EJB for the load-balancing and fail-over logic.

Cluster Map on the Client Side

The CMI cluster stub handles a cluster map on the client side:

• at lookup time for the SSB, SFSB, EB Home objects if the retrieved object is replicated and located in the global registry
• at create time for the SSB Remote objects if the retrieved object is replicated and located in the global registry

In these two cases, the call gets a stubs list from the global registry and the CMI cluster stub updates the local cluster map.

If a communication error with a server occurs during a remote call invocation, the server is removed from the cluster map.

CMI Class

CMI classes are generated by GenIC when compiling with the protocol CMI. They are built from the velocity templates located in $JONAS_ROOT/templates/genic directory. By default the templates used are:

• ClusterHomeDistributor.vm for the home interface of the SSB, SFSB, EB
• ClusterRemoteSLSBDistributor.vm for the remote interface of the SSB

The templates inherit the org.objectweb.carol.cmi.Distributor class and contain the following methods:

• choose(method, params): this method is called by the CMI cluster stub at each remote method call to choose the next stub to use for invoking the call. By default, this method implements the weighted round-robin algorithm with a local preference, i.e if there is a collocated server, it will be selected in priority. The servers factors are set for each JOnAS instance in the $JONAS_BASE/conf/carol.properties file.
• onException(method, params, stub, exception): this method is called by the CMI cluster stub when an exception occurs and returns a decision of RETRY, RETURN, or THROW for indicating the behavior to the CMI smart stub. By default, this method returns a RETRY decision if the exception is related to a network error, and otherwise it returns a THROW decision.
• onReturn(method, params, stub, returnValue): this method is called by the CMI cluster stub after having invoked the call and before returning to the application. This call back enables implementation of a post-processing on the return value. By default, this method does nothing.

Customizing the Load-balancing and Fail-over Logic

The user has the ability to customize the load-balancing and fail-over logic for each EJB by specifying the velocity template to use in the JOnAS- specific descriptor on deployment of the ejb-jar file. The XML elements are:

<cluster-home-distributor>MyHomeDistributor.vm</cluster-home-distributor>

<cluster-remote-distributor>MyRemoteDistributor.vm</cluster-remote-distributor>

If not set, the default velocity templates are used.

If set with the value 'disabled', the CMI classes are not generated and the EJB will not be distributed.

If set with a file name, this file must be located in the $JONAS_ROOT/templates/genic directory.

The 'cluster-home-distributor' element is valid for the SSB, SFSB and EB.

The 'cluster-remote-distributor' element is valid for the SSB.

SFSB High Availability

Stateful session beans (SFSBs) can be replicated in JOnAS 4.7 in order to provide high availability in the case of failures in clustered environments. A new service called High Availability (HA) has been included in JOnAS to support replication. JOnAS replication also requires the cluster method invocation (CMI) protocol.

Replication Description

JOnAS implements an update-everywhere replication protocol according to the database replication terminology (See the J. Gray et al.'s paper ''The dangers of replication and a solution'' in proceedings of the ACM SIGMOD 96's conference in Canada). In this protocol, a client can connect to any server. When the client calls the create() method on the SFSB's Home interface, the server the client connects to is selected following a round-robin scheme. All the requests from the client to the SFSB will be processed by this server until the client calls the remove() method on the remote interface. The rest of the servers will act as backups for that client. Before sending the reponse to the client, the SFSB's state is sent to the backups.

If the server fails, another server among the backups will be selected to serve the client requests, first restoring the current state of the SFSBs from the state information stored in the HA local service. From this point on, this server will receive the new client requests.

Concerning the SFSB's state that is replicated in the current version (JOnAS 4.7), if local or remote references to other types of EJBs are included in the state, only the remote references to other SFSBs are replicated.

The supported replication scenarios are shown in the following figure:

Configuring JOnAS for SFSB Replication

The High Availability (HA) service is required in JOnAS in order to replicate SFSBs. The HA service must be included in the list of available services in JOnAS. This is done in the jonas.properties file placed in $JONAS_BASE/conf.

...
jonas.services registry,jmx,jtm,db,dbm,security,resource,ejb,ws,web,ear,ha
...

The HA service must be configured also in the jonas.properties file:

...
jonas.service.ha.class org.objectweb.jonas.ha.HaServiceImpl
jonas.service.ha.gcl jgroups
...

The HA service uses JGroups as a group communication layer (GCL). JGroups behavior is specified by means of a stack of properties configured through a XML file (See JGroups documentation for more information:http://www.jgroups.org). The default configuration of the HA service uses the $JONAS_BASE/conf/jgroups-ha.xml file. You can tell the HA service to use a particular stack configuration by modifying the following line in jonas.properties :

...
jonas.service.ha.jgroups.conf jgroups-ha.xml
...

...
carol.protocols=cmi...
...

Configuring SFSBs for Replication

In order to replicate an SFSB, it is necessary to add the <cluster-replicated/> element to the SFSB definition in the jonas-ejb-jar.xml deployment descriptor. This element can have two possible values: true or false (default value). In addition, if the programmer wants to change the behaviour of the CMI stubs (e.g. the server selection policy), it is possible to specify different distributor implementations by means of <cluster-home-distributor/> and <cluster-remote-distributor/> elements. In this case, the value corresponds to the .vm file that implements the distributor in its home and remote parts respectively. If the <cluster-replicated/> element is present without the <cluster-*-distributor/> elements, the default values are used (ClusterHomeSFSBRepDistributor.vm and ClusterRemoteSFSBRepDistributor.vm).

The following is an example description for a replicated SFSB in jonas-ejb-jar.xml file:

...
<jonas-session>
   <ejb-name>DummySFSB</ejb-name>
   <jndi-name>DummySFSB</jndi-name>
   ...
   <cluster-replicated>true</cluster-replicated>
   <cluster-home-distributor>Dummy_HomeDistributor.vm</cluster-home-distributor>
   <cluster-remote-distributor>Dummy_RemoteDistributor.vm</cluster-remote-distributor>
</jonas-session>
...

Finally, when compiling the application that includes the replicated beans, the CMI protocol must be specified in order to generate the classes that include the replication logic.