Ideal Test Plan


Topics for Test Plans

Aims

Testing, verification, validation are related to the concepts of quality.

The aim of this page is to provide an ideal test plan for JOnAS in order to measure the software quality level and to contribute to improve it.

We consider that the more tests are automatics and integrated in the continuous integration process, the better they are.

We address here only the System Test (not the unit tests) and we focus on functional testing and non-functional testing as described below.

Type of tests

Functional testing

Functional testing refers to test all features and functions of a system to ensure requirements and specifications are met. 

Performance Testing, Load Testing, Stability Testing

Performance testing is executed to determine how fast a system performs under a particular workload.

Load testing is primarily concerned with testing that can continue to operate under a heavy load (stress).

Stability testing checks to see if the software can continuously function well in or above a long period.

These non-functional testing can also serve to validate and verify other quality attributes of the system, such as scalability, reliability. The resource usage must be monitored.

Targeted JOnAS test coverage

JavaEE Specifications

Functional testing of the JavaEE Specifications is done by using the Compatibility Test Suite for JavaEE 5 which is a complete functional test suite composed of more than 27000 tests.

Commands

  • jonas - jclient - newjb - newjc - raconfig - ...

Automatic functional tests of those commands have to be done on various platforms (Unix/Linux, Windows, Mac).

Administration Console

  • Administration Consoles Desktop
  • JOnAS Admin
  • New generation console modules: "JDBC Connection leaks"
  • other consoles coming from JASMINe-*

Automatic functional tests of those consoles have to be done. Note that those consoles use two different technologies: Struts and Flex.

Administration API

  • JMX
  • MEJB

Development Tools

  • JOPE: plugin for Eclipse
  • JOnbAS: Plugin NetBeans
  • Cargo
  • ant tasks
  • maven plugins

Automatic functional tests of those tools have to be done.

Deployment and dynamicity

JOnAS provides

  • advanced deployment functionalities (java EE artefacts, OSGI bundles, deployment plan, ...)
  • and dynamicity with the JOnAS services on demand

Those functionalities have to be test automatically.

Versioning

The JOnAS service versioning allows to upgrade the version of an application without service interruption.

Extensions

The JOnAS 5 platform is extensible and some SOA tools such as JBoss rules engine (drools) or Apache Camel framework were integrated and specific test suites can be added.

Interoperability

Web services interoperability testing with other Application Servers can be done. 

OSGi Platform

  • Automatic functional tests have to be done about the life cycle. Testing the starting/stopping of the whole JVM or only a subset of services. Checking the elapsed time.
  • Evaluate the limits of the JOnAS OSGI server in term of number of bundles and number of services

Enterprise OSGi

JOnAS 5 is an entreprise OSGi application servet. The http service in delivered in a first step. A new test suite has to be developed for covering these aspects.

Clustering

JOnAS provides an end to end solution for clustering that ensures transparent distribution, high availability and scalability of Java EE applications.

  • At the web level through
    • Apache/mod_jk or Apache/mod_proxy_balancer for load-balancing the HTTP flow between multiple JOnAS/Tomcat instances.
    • Tomcat for providing a TCP-based solution of HTTP session replication, ensuring their high availability
  • At the EJB level (EJB2 and EJB3 support) through CMI and its cluster proxy
  • At the JMS level through JORAM server and JORAM HA.

Various performance testing topics:

  • scalability of the system: benefit to have severals web servers, ejb servers, jms servers and evaluation of the maximum number of servers
  • high availability of the system: measure the cost of the execution of
    • http session replication
    • ejb session replication

Repeatable Performance testing focus on various topics

Repeatable performance testing focus on various topics:

  • webapp: JOnAS/Tomcat, JOnAS/Jetty
  • webservice: JOnAS/CXF, JOnAS/Axis
  • jms: JOnAS/JORAM
  • ejb: JOnAS/EasyBeans

to get an idea about the JOnAS performance in terms of webservices calls per second, JMS message treatment per second, .... for each JOnAS version.

Assets

Existing tests

EasyBeans

Easybeans main modules contain functional testing included in the maven build process, and so included in the Easybeans SVN repository. See the tests in the following modules easybeans, easybeans-osgi, ow2-bundles, spec and util in http://websvn.ow2.org/listing.php?repname=easybeans&

Those tests are executed on continuous

JOnAS

Unlike Easybeans, just only few tests are included in the JOnAS maven build process. Most of them are included in the jonas_tests part of in the JOnAS SVN repository. See http://websvn.ow2.org/listing.php?repname=jonas&path=%2Fjonas%2Ftrunk%2F#path_jonas_trunk_

Those tests are executed on continuous

On Bamboo, same tests are launch severals times on various environments (JDK 1.5, JDK 1.6, protocols JRMP or CMI or IIOP).

Conformance Test Suites for J2EE 1.4 and JavaEE 5

Two Conformance Test Suites, respectively for J2EE 4 and JavaEE 5, are used. Specific tools and environments for JOnAS 4 and for JOnAS 5 allow to configure and run those tests automatically. Those tools are archived in the OW2 SVN Repository which got limited access right. Using rights of those CTS are strictly limited.

Standards bench of SPEC

Two standard benchs of J2EE 1.4 and JavaEE 5 are run by the JOnAS Team:

For the SPECJAPPServer2004 bench, we got specific tools to help us to easier run this bench on JOnAS 4 and 5. Those tools are archived in a private Bull forge in the jemval project: SPECj2004 module. (See --http://licorne.frec.bull.fr/plugins/scmsvn/viewcvs.php/trunk/SPECj2004/?root=jemval--) Equivalent has to be done for the new SPECjEnterprise2010 bench.

Miscellaneous

We got on a private Bull forge, various applications and environments for performance testing: --http://licorne.frec.bull.fr/plugins/scmsvn/viewcvs.php/trunk/SPECj2004/?root=jemval--.

  • jappli-simplebank2: simple bank application (debit and credit bank account) on EJB2
  • jappli-simplebank2: simple bank application on EJB3
  • jpa-jazzcat: a catalog of jazz recordings used
  • jemval-jpa: performance testing of various jpa implementations (use of jpa-jazzcat application)
  • jemval-jonas: performance testing of web containers (tomcat/jetty).

Used tools

Bamboo and Hudson

Bamboo and Hudson are the tools used for the continuous integration.

Junit, HttpUnit, and TestNG

Junit, HttpUnit, and TestNG are the testing framework used. For Flex console, Flexunit might be considered. For OSGi platform, Pax Exam or Junit4OSGi might be considered.

CLIF

For performance testing, the used tool to generate the load is OW2 CLIF. See http://clif.ow2.org/

Contribution items

This table summarizes the main items which can be achieved by JOnAS community contributors.

ItemContributorsPriority
Admin: Automatic tests of the new generation consoles (flex)?++
OSGI Platform: lifecycle bundles and services?++
OSGI Platform: servlet deployment on the HTTP OSGi service?+
Clustering: scalability of a JOnAS cluster and limits?+
Performance webappBull,FT+
Performance web services?++
Performance jmsBull+
Performance ejb?++