Ideal Test Plan
Topics for Test Plans
Testing, verification, validation are related to the concepts of quality.
The aim of this page is to provide an ideal test plan for JOnAS in order to measure the software quality level and to contribute to improve it.
We consider that the more tests are automatics and integrated in the continuous integration process, the better they are.
We address here only the System Test (not the unit tests) and we focus on functional testing and non-functional testing as described below.
Type of tests
Functional testing refers to test all features and functions of a system to ensure requirements and specifications are met.
Performance Testing, Load Testing, Stability Testing
Performance testing is executed to determine how fast a system performs under a particular workload.
Load testing is primarily concerned with testing that can continue to operate under a heavy load (stress).
Stability testing checks to see if the software can continuously function well in or above a long period.
These non-functional testing can also serve to validate and verify other quality attributes of the system, such as scalability, reliability. The resource usage must be monitored.
Targeted JOnAS test coverage
Functional testing of the JavaEE Specifications is done by using the Compatibility Test Suite for JavaEE 5 which is a complete functional test suite composed of more than 27000 tests.
- jonas - jclient - newjb - newjc - raconfig - ...
Automatic functional tests of those commands have to be done on various platforms (Unix/Linux, Windows, Mac).
- Administration Consoles Desktop
- JOnAS Admin
- New generation console modules: "JDBC Connection leaks"
- other consoles coming from JASMINe-*
Automatic functional tests of those consoles have to be done. Note that those consoles use two different technologies: Struts and Flex.
- JOPE: plugin for Eclipse
- JOnbAS: Plugin NetBeans
- ant tasks
- maven plugins
Automatic functional tests of those tools have to be done.
Deployment and dynamicity
- advanced deployment functionalities (java EE artefacts, OSGI bundles, deployment plan, ...)
- and dynamicity with the JOnAS services on demand
Those functionalities have to be test automatically.
The JOnAS service versioning allows to upgrade the version of an application without service interruption.
The JOnAS 5 platform is extensible and some SOA tools such as JBoss rules engine (drools) or Apache Camel framework were integrated and specific test suites can be added.
Web services interoperability testing with other Application Servers can be done.
- Automatic functional tests have to be done about the life cycle. Testing the starting/stopping of the whole JVM or only a subset of services. Checking the elapsed time.
- Evaluate the limits of the JOnAS OSGI server in term of number of bundles and number of services
JOnAS 5 is an entreprise OSGi application servet. The http service in delivered in a first step. A new test suite has to be developed for covering these aspects.
JOnAS provides an end to end solution for clustering that ensures transparent distribution, high availability and scalability of Java EE applications.
- At the web level through
- Apache/mod_jk or Apache/mod_proxy_balancer for load-balancing the HTTP flow between multiple JOnAS/Tomcat instances.
- Tomcat for providing a TCP-based solution of HTTP session replication, ensuring their high availability
- At the EJB level (EJB2 and EJB3 support) through CMI and its cluster proxy
- At the JMS level through JORAM server and JORAM HA.
Various performance testing topics:
- scalability of the system: benefit to have severals web servers, ejb servers, jms servers and evaluation of the maximum number of servers
- high availability of the system: measure the cost of the execution of
- http session replication
- ejb session replication
Repeatable Performance testing focus on various topics
Repeatable performance testing focus on various topics:
- webapp: JOnAS/Tomcat, JOnAS/Jetty
- webservice: JOnAS/CXF, JOnAS/Axis
- jms: JOnAS/JORAM
- ejb: JOnAS/EasyBeans
to get an idea about the JOnAS performance in terms of webservices calls per second, JMS message treatment per second, .... for each JOnAS version.
Easybeans main modules contain functional testing included in the maven build process, and so included in the Easybeans SVN repository. See the tests in the following modules easybeans, easybeans-osgi, ow2-bundles, spec and util in
Those tests are executed on continuous
- on the Bamboo OW2 continue integration environment on a Linux platform: See
- on a Bull private Hudson continue integration environment on a windows platform: See --
Unlike Easybeans, just only few tests are included in the JOnAS maven build process. Most of them are included in the jonas_tests part of in the JOnAS SVN repository. See
Those tests are executed on continuous
- on the Bamboo OW2 continuous integration environment on a Linux platform: See for most of them
- on the private Hudson continuous integration environment on a windows platform for the cluster test: --
On Bamboo, same tests are launch severals times on various environments (JDK 1.5, JDK 1.6, protocols JRMP or CMI or IIOP).
Conformance Test Suites for J2EE 1.4 and JavaEE 5
Two Conformance Test Suites, respectively for J2EE 4 and JavaEE 5, are used. Specific tools and environments for JOnAS 4 and for JOnAS 5 allow to configure and run those tests automatically. Those tools are archived in the OW2 SVN Repository which got limited access right. Using rights of those CTS are strictly limited.
Standards bench of SPEC
Two standard benchs of J2EE 1.4 and JavaEE 5 are run by the JOnAS Team:
For the SPECJAPPServer2004 bench, we got specific tools to help us to easier run this bench on JOnAS 4 and 5. Those tools are archived in a private Bull forge in the jemval project: SPECj2004 module. (See --Equivalent has to be done for the new SPECjEnterprise2010 bench.
We got on a private Bull forge, various applications and environments for performance testing: --
- jappli-simplebank2: simple bank application (debit and credit bank account) on EJB2
- jappli-simplebank2: simple bank application on EJB3
- jpa-jazzcat: a catalog of jazz recordings used
- jemval-jpa: performance testing of various jpa implementations (use of jpa-jazzcat application)
- jemval-jonas: performance testing of web containers (tomcat/jetty).
Bamboo and Hudson
Bamboo and Hudson are the tools used for the continuous integration.
Junit, HttpUnit, and TestNG
Junit, HttpUnit, and TestNG are the testing framework used. For Flex console, Flexunit might be considered. For OSGi platform, Pax Exam or Junit4OSGi might be considered.
For performance testing, the used tool to generate the load is OW2 CLIF. See
This table summarizes the main items which can be achieved by JOnAS community contributors.
|Admin: Automatic tests of the new generation consoles (flex)||?||++|
|OSGI Platform: lifecycle bundles and services||?||++|
|OSGI Platform: servlet deployment on the HTTP OSGi service||?||+|
|Clustering: scalability of a JOnAS cluster and limits||?||+|
|Performance web services||?||++|