CMI (Clustered Method Invocation)

Introduction

CMI is the protocol cluster for JOnAS ensuring:

CMI architecture

Getting started with the CMI Protocol

CMI can be enabled in JOnAS by:

Clustered objects and CMI Registry

CMI brings its own registry for implementing the jndi replication. Each CMI registry instance over the cluster contains two areas:

When an object is registered in the registry, the routing to the local or global area is done according to:

The entries of the distributed area are lists providing the ability, for example, to gather several stubs for the the same jndi-name and thus to return a stubs list.

JNDI HA

Registry Replication

CMI relies on JGroups group-communication protocol for ensuring the global registry replication. The parameters are gathered in the:

CMI registry

All the members of a cluster share the same JGroups configuration.

If several cluster partitions are required over a single LAN, several JGroups configurations must be configured with different values for the following parameters:

When a new node appears in the cluster, its registry content is synchronized automatically.

When a node disappears, JGroups notifies the other's member of the node leaving and the registry entries related to this node are removed.

Registry Fail-over

On the client side, the high availability of the registry is provided by the capability to set several JOnAS instances in the registry url. At the lookup time, the client chooses (round-robin algorithm) one of the available servers to get the home stub. If the server fails, the request is sent to another server. The CMI url registry is specified in the $JONAS_BASE/conf/carol.properties file using the following syntax:

carol.cmi.url=cmi://server1:port1[,server2:port2...] 

CMI Cluster Stub or Cluster-aware Stub

Load-balancing and fail-over on the client side are provided through cluster-aware stubs. These stubs are generated on the fly through ASM and rely on:

CMI cluster stub

Cluster Map on the Client Side

The CMI cluster stub handles a cluster map on the client side. The CMI cluster stub is created:

In these two cases, the call gets a stubs list from the global registry and the CMI cluster stub updates the local cluster map. Afterwards, the local cluster map can be updated dynamically during the invocation of the business methods calls (through the HA interceptors) when a new view is detected in the cluster.

If a communication error with a server occurs during a remote call invocation, the server is removed from the cluster map.

CMI Class

CMI classes are generated by GenIC when compiling with the protocol CMI. They are built from the velocity templates located in $JONAS_ROOT/templates/genic directory. By default the templates used are:

The templates inherit the org.objectweb.carol.cmi.Distributor class and contain the following methods:

Customizing the Load-balancing and Fail-over Logic

The user has the ability to customize the load-balancing and fail-over logic for each EJB by specifying the velocity template to use in the JOnAS- specific descriptor on deployment of the ejb-jar file. The XML elements are:

<cluster-home-distributor>MyHomeDistributor.vm</cluster-home-distributor>
<cluster-remote-distributor>MyRemoteDistributor.vm</cluster-remote-distributor>

If not set, the default velocity templates are used.

If set with the value 'disabled', the CMI classes are not generated and the EJB will not be distributed.

If set with a file name, this file must be located in the $JONAS_ROOT/templates/genic directory.

The 'cluster-home-distributor' element is valid for the SSB, SFSB and EB.

The 'cluster-remote-distributor' element is valid for the SSB.

High Availability with Horizontal Replication

Stateful session beans (SFSBs) can be replicated since JOnAS 4.7 in order to provide high availability in the case of failures in clustered environments. A new service called High Availability (HA) has been included in JOnAS to provide replication mechanishms. JOnAS HA also requires the cluster method invocation (CMI) protocol.

Compared to JOnAS 4.7, JOnAS 4.8 implements a new replication algorithm based on a horizontal replication approach. The algorithm improves the algorithm implemented for JOnAS 4.7 with the following enhancements:

  • Replication of SFSBs with references to EBs: The algorithm can replicate SFSBs that reference EB by means of both, local or remote interfaces.
  • Transaction awareness: The algorithm is transaction aware, meaning that the state is not replicated if the transaction aborts.
  • Exactly-once semantics: Each transaction is committed exactly once at the DB if the client does not fail. If the client fails, each transaction is committed at most once at the DB
  • EJB replication Description

    Update-everywhere mode

    JOnAS implements an update-everywhere replication protocol according to the database replication terminology (See the J. Gray et al.'s paper ''The dangers of replication and a solution'' in proceedings of the ACM SIGMOD 96's conference, Canada). In this protocol, a client can connect to any server. When the client calls the create() method on the SFSB's Home interface, the server the client connects to is selected following a round-robin scheme. All the requests from the client to the SFSB will be processed by this server until the client calls the remove() method on the remote interface. The rest of the servers will act as backups for that client. Before sending the response to the client, the SFSB's state is sent to the backups.

    If the server fails, another server among the backups will be selected to serve the client requests, first restoring the current state of the SFSBs from the state information stored in the HA local service. From this point on, this server will receive the new client requests.

    The supported replication scenarios are shown in the following figure:

    Replication scenarios in JOnAS 4.8

    Transaction aware fail-over

    The horizontal approach aims to guarantee that the transactions are kept consistent when a fail-over occurs. They are either aborted or restored for ensuring the exactly-once semantics. During a fail-over, the new primary uses a special table in the database for storing the transaction identifier and enabling to find out if the transaction was committed or not.

    Beyond the SFSB replication, the algorithm enables the building of applications (stateful or stateless) with a high level of reliability and integrity.

    Configuring JOnAS for EJB Replication

    The High Availability (HA) service is required in JOnAS in order to replicate SFSBs. The HA service must be included in the list of available services in JOnAS. This is done in the jonas.properties file placed in $JONAS_BASE/conf.

    ...
    jonas.services registry,jmx,jtm,db,dbm,security,resource,ejb,ws,web,ear,ha
    ...

    The HA service must also be configured in the jonas.properties file:

    ...
    jonas.service.ha.class org.objectweb.jonas.ha.HaServiceImpl
    jonas.service.ha.gcl jgroups
    ...
    

    The HA service uses JGroups as a group communication layer (GCL). JGroups behavior is specified by means of a stack of properties configured through an XML file (See JGroups documentation for more information: http://www.jgroups.org). The default configuration of the HA service uses the $JONAS_BASE/conf/jgroups-ha.xml file and the sfsb-rep group name. The HA service can be told to use a particular stack configuration or a particular group name by modifying the following lines in jonas.properties:

    ...
    jonas.service.ha.jgroups.conf jgroups-ha.xml
    jonas.service.ha.jgroups.groupname jonas-rep
    ...
    Finally, the CMI protocol must be specified in the carol.properties file in $JONAS_BASE/conf:
    ...
    carol.protocols=cmi...
    ...

    Transaction Table Configuration

    The new horizontal replication algorithm uses a database table to keep track of current running transactions. This table is accessed from the new elected node during fail-over to detect whether or not the current transaction committed at the former local node, ensuring exactly-once semantics. The table contains only one column: the transaction identifier (txid).

    In JOnAS 4.8 this table must be created manually with the following SQL command:

    create TABLE ha_transactions (txid varchar(60));

    This table should be located preferably in the database used by the replicated application, but it is not mandatory. If the table is not created in the database used by the replicated application, it is necessary to configure a new datasource for the database that contains this transaction table. This datasource must be configured to use the serializable transaction isolation level.

    The database that holds the transaction table is accessed by the replication service with the JNDI name configured in jonas.properties.

    ...
    jonas.service.ha.datasource tx_table_ds
    ...

    Configuring Garbage Collection

    Due to the fact that the replication algorithm stores information associated with clients' transactions and that the server is not notified when a client dies, the HA service might have been storing unnecessary replication information over time. In order to automatically clean this unnecessary replication information, the HA service includes a garbage collection mechanism. It is possible to configure the number of seconds the system waits to execute this mechanism by changing the following property in the jonas.properties file:
    ...
    jonas.service.ha.timeout 600
    ...

    Configuring an Application for Replication

    jonas-ejb-jar.xml

    In order to configure an application for replication, the <cluster-replicated/> element must be added to the bean definition of every bean requiring high availability in the jonas-ejb-jar.xml deployment descriptor file. This element can have two possible values: true or false (default value). In addition, if the programmer wants to change the behavior of the CMI stubs (e.g., the server selection policy), it is possible to specify different distributor implementations by means of <cluster-home-distributor/> and <cluster-remote-distributor/> elements. In this case, the value corresponds to the .vm file that implements the distributor in its home and remote parts respectively. If the <cluster-replicated/> element is present without the <cluster-*-distributor/> elements, the default values are used (ClusterHomeSFSBRepDistributor.vm and ClusterRemoteSFSBRepDistributor.vm).

    The following is an example description for a replicated SFSB in jonas-ejb-jar.xml file:

    ...
    <jonas-session>
       <ejb-name>DummySFSB</ejb-name>
       <jndi-name>DummySFSB</jndi-name>
       ...
       <cluster-replicated>true</cluster-replicated>
       <cluster-home-distributor>Dummy_HomeDistributor.vm</cluster-home-distributor>
       <cluster-remote-distributor>Dummy_RemoteDistributor.vm</cluster-remote-distributor>
    </jonas-session>
    ...

    The <cluster-replicated/> element can also be set in the SSB or EB for

    Note: When set in the SSB, the mechanism inhibits the load-balancing at the remote interface. After the home create() method call, all the requests are sent to the same instance.

    Entity Beans lock policy

    The lock policy for the Entity Beans in a replicated application must be configured as database in the jonas-ejb-jar.xml deployment descriptor file.

    The following is an example description for a replicated EB in the jonas-ejb-jar.xml:

    ...
    <jonas-entity>
        <ejb-name>MyEntitySLR</ejb-name>
        <jndi-name>MyEntityHome</jndi-name>
        <cluster-replicated>true</cluster-replicated>
        <shared>true</shared>
        <jdbc-mapping>
            <jndi-name>example_ds</jndi-name>
        </jdbc-mapping>
        <lock-policy>database</lock-policy>
    </jonas-entity>
    ...

    Datasource used by the application

    The datasources used by replicated applications must be configured to use the serializable transaction isolation level.

    The following is an example for a datasource configuration file for the Postgres DBMS:

    ...
    datasource.name         example_ds
    datasource.url          jdbc:postgresql://xxx.xxx.xxx.xxx:xxxx/database
    datasource.classname    org.postgresql.Driver
    datasource.username     jonas
    datasource.password
    datasource.mapper       rdb.postgres
    datasource.isolationlevel       serializable
    ...

    Finally, when compiling the application that includes the replicated beans, the CMI protocol must be specified in order to generate the classes that include the replication logic.

    Status and Management Information in Admin. Console for the HA Service's Replication Algorithm

    The JOnAS administration console offers several items of information about the HA service's replication algorithm and allows the configuring of several parameters related to its behaviour. The related information and parameters include: