Internally, we have what we call the tech watch day where we investigate a library and/or a technology during a day with a concrete objective.  This article aims to explain what we have tried to integrate custom rules in Sonar. We choose CheckStyle because we have quickly found documentation on how to write rules and there is also a nice IntelliJ plugin to run rules inside the IDE. We do not pretend it was the best option, but a first trial.

CheckStyle uses its own API on top of AST. The analysis is based on the source code and is limited to the class itself. It means that we cannot create a constraint based on the source code of multiple classes. For instance, if a class A must implement an interface I, A must directly implement this interface; A extending B where B implements I is not supported. This is a major drawback, but for the rules we intended to implement this was not blocking.

The project hosted on github has the following modules:

• A reusable tester for CheckStyle rules,
• Some CheckStyle utilities,
• One sample implementation of a CheckStyle rule that does not depend on Sonar at all,
• A module that generates a  sonar extension plugin based on that rule,
• A integration test that runs a sample project against the created rule.

Checkstyle Tester

This module provides a tester performing a CheckStyle analysis inside a (unit) test. After that it helps to easily write assertions on the result. The example below shows how the rule UnusedImportsCheck provided out-of-the-box by CheckStyle could be tested. Assume that we run this rule against a class that imports java.util.Date and does not use it. The unused import is located at line 3, column 8.

@Test
public void classWithUnusedImport() {
    getAnalysis().get(ClassWithUnusedImport.class).assertOnlyThose(
        violation(UnusedImportsCheck.class)
            .on(3, 8)
            .withMessage("Unused import - java.util.Date.")
            .withLevel(SeverityLevel.ERROR)
    );
}

The tester also allows to test localized messages. In the snippet below we test the French message:

@Test
public void classWithUnusedImportFrench() {
   final CheckStyleTester analysis =          
      CheckStyleTesterBuilder.forConfigFile("/checkstyle-config.xml")
         .withSourceLocationAsProperty().withLocale(Locale.FRENCH).build().assertNoException();

   analysis.get(ClassWithUnusedImport.class).assertOnlyThose(
      violation(UnusedImportsCheck.class)
         .on(3, 8)
         .withMessage("Import inutilisé - java.util.Date.")
         .withLevel(SeverityLevel.ERROR)
   );
}

Custom Rule

Now we are able to test a CheckStyle rule, it is about time to create a new rule using some of the utilities exposed in the com.bsb.common.integration.checkstyle-utils module. Let’s implement a rule (ImplementClausesCounterCheck) that checks that a given class does not implement more than two interfaces. Since we have a single potential violation, let’s extend from AbstractSingleCheck. An instance of the check class is created once per analysis regardless of the number of classes to be analysed. The check class is called back when a Java file starts to be processed and when the analysis on this file is finished. There is also a callback when a tree node of interest is encountered. A node can be the declaration of a class, method, or a statement like an implements, extends, etc.

In this case, it means that when a class definition is found, the rule is called back. The rule simply asks for the implements statement and the number of identifiers in it, that is the number of implemented interfaces. If the maximum is reached, a message is logged.

@Override
public void finishTree(DetailAST def) {
    super.finishTree(def);

    if (!isConstraintSatisfied()) {
        log(getViolationLine(), getViolationColumn(), getViolationMessage());
    }
}

The rule is tested against a set of sample classes representing various use cases: one with no implement, one with a number of implements just before the maximum, a class with a violation and a class with an inner class. There are also translation tests that validates the proper message is set in English and French.

This project is also configured to generate a JAR file containing the project and its dependencies. Its purpose is to be usable by the IntelliJ plugin checkstyle-idea. The screenshots bellow describe how to configure the plugin.

Sonar Plugin

The CheckStyle rule can be now exposed as a Sonar plugin. The following description is based on the Sonar documentation page. An XML file contains the declaration of the new Sonar rule (description, id, severity, etc). The class CheckStyleExtensionRepository is responsible of providing Sonar rules. The repository loads the XML file and loads the CheckStyle rule as a Sonar rule. CheckStyleExtensionPlugin declares the extension and specifies the provider to use.

The POM file of this module can install the extension in your Sonar instance. For that, it expects that the directory containing your Sonar installation can be accessed by the file system. The command simply copies the JAR to the path specified in the main POM file (see the sonar.dir property).

You can either change the value of sonar.dir in the POM itself or override it on the command line. For instance, to install the plugin in c:\tools\sonar:

bsblabs@bsblabs ~/sonar-rule-showcase/samples/com.bsb.samples.sonar.implements-checker-plugin
$ mvn clean install -PinstallPlugin -Dsonar.dir=c:\tools\sonar

Sonar Integration Tests Module

The integration tests module (not part of the Maven reactor) is configured to analyse the samples written to test the rule in Sonar itself. Of course it expects that the Sonar extension has been previously deployed. The properties that Sonar need are located in the POM file. The analysis is simply performed during the package phase:

bsblabs@bsblabs ~/sonar-rule-showcase/samples/com.bsb.samples.sonar.implements-checker-plugin-its
$ mvn clean package

During this phase, the Maven Checkstyle Plugin also performs an analysis. This analysis is completely independent from the Sonar analysis. The only purpose is to show how a new rule can be tested by this plugin. The generated reports are available in the target directory of that module. Of course, the report contains the same violations as the ones displayed by Sonar.

Note that this automated process is optional. When you want to quickly test a rule that you are changing this process might be handy. But you can of course copy the plugin and analyse one of your projects through the Sonar interface as well.

Conclusion

This article briefly describe how one could implement and test a CheckStyle rule and define it as a Sonar plugin. The project available on github describes the different steps to achieve that goal and contains reusable classes to write and test CheckStyle rules.

We already has similar problems when migrating from Maven 2 to Maven 3, due to the new library used by Maven to access the repository system: Aether. These problems only came when using plugins that had to resolve dependencies by themselves.

Running these plugins with Maven 2 require the use of the org.apache.maven.artifact.resolver package (ArtifactResolver and ArtifactCollector classes, a. o.). Although still valid when running the plugin with Maven 3, this can lead to some side effects we experienced, such as:

• Dependencies explicitly marked as excluded still being resolved
• Dependencies taken from a Maven 1 repository being unresolvable because coordinates stated as null:null:jar (seems specific to Maven 3.0.4+)
• Inconsistencies with the dependency mechanism used internally by Maven (see here)

So we had to update our plugins to use the new Aether library to perform dependency resolution, and found a good starting point on the Sonatype blog.

Let’s take one sample method that would need to be updated. For instance, the following method is responsible of resolving all dependencies of a project:

public static Set<Artifact> getDependencyArtifacts(MavenProject project, ArtifactResolver resolver, ArtifactRepository localRepository, 
            List<?> remoteRepositories, ArtifactMetadataSource artifactMetadataSource) throws MojoExecutionException {
    ArtifactResolutionResult resolutionResult;
    try {
        resolutionResult = resolver.resolveTransitively(project.getArtifacts(), project.getArtifact(), remoteRepositories, 
                localRepository, artifactMetadataSource);
    } catch (ArtifactResolutionException e) {
        throw new MojoExecutionException(e.getMessage(), e);
    } catch (ArtifactNotFoundException e) {
        throw new MojoExecutionException(e.getMessage(), e);
    }
    return resolutionResult.getArtifacts();
}

This is quite a simple process, as all needed components (project, resolver, localRepository, remoteRepositories and artifactMetadataSource) can be injected by Plexus, either by an expression or a component “annotation”:

/**
     * @parameter expression="${project}"
     * @required
     * @readonly
     */
    protected MavenProject project;

    /**
     * Local repository to be used by the plugin to resolve dependencies.
     * @parameter expression="${localRepository}"
     */
    protected ArtifactRepository localRepository;

    /**
     * List of remote repositories to be used by the plugin to resolve dependencies.
     * @parameter expression="${project.remoteArtifactRepositories}"
     */
    protected List remoteRepositories;

    /**
     * @component
     */
    protected ArtifactResolver resolver;

    /**
     * @component
     */
    protected ArtifactMetadataSource artifactMetadataSource;

As already said, this code can still be executed with Maven 3, thanks to the great job the Maven guys did in making it globally backward compatible. But this sometimes fails to do the job, and this is not the way we should write new plugins!

The Sonatype blog post is quite nice, but using it as is would mix a lot of the same concepts. Yes, part of the classes you manipulate to handle dependency resolution are both in Aether and Maven API, notably Maven Artifact and Aether Artifact. A plugin will usually work on the Maven Artifact side, so we rather want to use the Maven packages as much as possible, with ideally no explicit reference about the underlying library that Maven uses. That would also minimize the modifications needed inside the plugin when upgrading to Maven 3.

Unfortunately, there’s no way to achieve dependency resolution without at least one import of an Aether class, but still this is valuable as it is limited to the dependency resolution process, not the dependencies or artifact structures, which is still nice to reduce the impact on the business part of the plugin work.

Here is the method modified to be compliant with Maven 3.0.4 and Aether 1.13.1

public static Set getDependencyArtifacts(MavenProject project, RepositorySystemSession repoSession, 
        ProjectDependenciesResolver projectDependenciesResolver) throws MojoExecutionException {

    DefaultDependencyResolutionRequest dependencyResolutionRequest = new DefaultDependencyResolutionRequest(project, repoSession);
    DependencyResolutionResult dependencyResolutionResult;

    try {
        dependencyResolutionResult = projectDependenciesResolver.resolve(dependencyResolutionRequest);
    } catch (DependencyResolutionException ex) {
        throw new MojoExecutionException(ex.getMessage(), ex);
    }

    Set artifacts = new LinkedHashSet();
    if (dependencyResolutionResult.getDependencyGraph() != null
            && !dependencyResolutionResult.getDependencyGraph().getChildren().isEmpty()) {
        RepositoryUtils.toArtifacts(artifacts, dependencyResolutionResult.getDependencyGraph().getChildren(), 
                Collections.singletonList(project.getArtifact().getId()), null);
    }
    return artifacts;
}

The only Aether class that we still have to use is RepositorySystemSession, all other are core Maven ones, especially the Artifact one. Also note that you do not have to depend on Aether explicitly, as it will be taken transitively from the org.apache.maven:maven-core:jar:3.0.4 dependency.

As for the previous implementation, the required parameters can be obtained with Plexus injections:

/**
     * @parameter expression="${project}"
     * @required
     * @readonly
     */
    protected MavenProject project;

    /**
     * The current repository/network configuration of Maven.
     *
     * @parameter default-value="{repositorySystemSession}"
     * @readonly
     */
    protected RepositorySystemSession repoSession;

    /**
     * @component
     */
    protected ProjectDependenciesResolver projectDependenciesResolver;

With an isolation of dependency resolution inside specific methods that will not have their return type changed, a plugin will easily be updated to support Maven 3 dependency resolution mechanism.
One drawback is the use of RepositoryUtils.toArtifacts, which is marked by Maven as intended to be used internally.

Similarly, resolving one artifact can be done like this:

public static void resolveArtifact(Artifact artifact, RepositorySystem repositorySystem, List remoteRepositories, 
        ArtifactRepository localRepository) throws MojoExecutionException {
    ArtifactResolutionRequest request = new ArtifactResolutionRequest()
        .setArtifact(artifact)
        .setRemoteRepositories(remoteRepositories)
        .setLocalRepository(localRepository);
    ArtifactResolutionResult resolutionResult = repositorySystem.resolve(request);
    if(resolutionResult.hasExceptions()){
        throw new MojoExecutionException("Could not resolve artifact: " + artifact, resolutionResult.getExceptions().get(0));
    }
}

This code does not depend directly on any Aether class.

Using the above samples, plugins can easily be migrated to fully comply with the new Maven 3 dependency resolution library.

What is code quality?

The first step is to define precisely what code quality means for us and how we can measure it. We decided to focus on these areas:

• Duplications (code copy-paste)
• Rules violations (static analysis)
• Code coverage
• Documentation (API)
• Package tangle
• Compiler warnings
• TODOs, FIXMEs and others reminders in code

From a tool standpoint, Jenkins can track compiler warnings and open tasks (TODO, etc) easily. Sonar can actually manage all the rest and much more.

So where do we go from there? Our team is using some of the agile methodologies, in particular the daily stand-up, iteration-based development, iteration review and retrospective. The end goal is not about measuring, it is about controlling how that measure evolves. Everything starts with an initial measure that gives an overview. Alongside the initial measure we need some guidelines to define what the objectives are and how to achieve them. These guidelines should improve over time and should adapt to new requirements.

At the end of an iteration (a development lifecycle), we measure the quality and we take a set of actions for the next one if not all quality criteria are met. Of course, this only works if those actions are known by the whole team and every single team member embraces them.

How do we measure code quality?

Measuring quality is about applying the concept of Continuous Inspection. Just like continuous integration makes sure your project builds and all tests pass, continuous inspection is about looking at your code and notify you when the quality decreases. Both Jenkins and Sonar have the ability to give a trend of what is going on over time.

The plugins we use in Jenkins allow to give an history of compiler warnings and open tasks:

The problem with those graphs is that you need to keep enough builds to get a proper history. By default (especially for the Maven project style), Jenkins stores a lot of data at each build but you can easily fix that by checking the Disable automatic artifact archiving advanced option of the build section in the project configuration. Once you fixed that, you still have the issue that Jenkins does not play nice if a project has a big job history since it seems to parse it when it starts. To fix that, you have to either discard old builds or keep only the latest X builds. In both cases, you’ll be back to the original problem since the history of your project will be lost when you reach that threshold. Luckily there is a solution. In our case, we want to eventually keep one build per iteration so we use the Keep this build forever action on the latest build of every iteration. That way the graph has a deeper history.

Sonar has a wonderful feature called the differential dashboard that basically allows you to diff two quality snapshots. Similarly to Jenkins, Sonar purges those quality snapshots according to a configurable policy. Sonar also allows you to name one particular quality snapshot either by assigning a version to it or an event. What we do is that at the end of each iteration, we assign the name of the iteration to the latest quality snapshot of that iteration.

To do so, go to the project in sonar and in the left menu, choose the History element under the Configuration section. Then choose the quality snapshot and click on the create button in the version column. Once you’ve done that, you can register that version so that it is available in the Time changes drop down list. To do so, go to Settings > Differential views and fill in the name of your version in one of the period fields.

Once we have configured Sonar that way, we are able to diff the current state of the project with the last or the before last iteration:

How do we improve code quality?

Properly configured tools are not enough to improve the overall quality of a project but a feature like the differential view in Sonar is definitely a very good leverage to achieve that goal. On top of these tools, we need Key Progress Indicator (KPIs), guidelines, rules and conventions. But most important of all, we need to do all this as a team. This is why we have built incrementally a team agreement page where we write down the things that we have openly discussed and agreed upon.

Concretely, we have the following code quality checks during the iteration:

• Jenkins mail notification (using the email-ext plugin) that provides a summary of what is going on: commits since the last build, tests that are failing and the proper excerpt of the console log
• Sonar notifications that are sent when new violations are introduced in the project
• A manual email and/or reminder at the stand-up that any team member can trigger if he feels something needs our attention
• The iteration retrospective where we look at Jenkins and Sonar to figure out together how the iteration went and determine if concrete actions are needed for the next iteration

If the quality objectives are not met, each team member has a concrete objectives for the next iteration: fix 10 items. An item is an generic concept: it defines one rule violation, one compiler warning, one open task, one package tangle, one copy-pasted block, 0.1% of code coverage or 0.1% of API documentation. Obviously, these may sound unfair and not well balanced (fixing a package tangle is probably much harder than fixing a simple rule violation) but on the other hand simple items never last until the end of the iteration. This gives a concrete objective to the team and nobody (including me as team lead) do care about who does what as long as we meet our team objective (6 team members = 60 items to fix).

There is also things that a tool will have a hard time to track: conventions. And code quality is amongst others about consistency. To make sure we are doing the same thing the same way, we have built a set of rules & conventions pages. These pages are also built incrementally and are based on concrete issues and/or inconsistency in the code. These are invaluable when we have to remember what we decided years ago or when a new team mate joins our team.

This process is also continuously improved: any  team member can write a proposal or request a change in the Sonar configuration. Based on a proposal that is fully linkable on our Wiki, we discuss it together at the stand-up, the iteration retrospective or in a specific meeting depending on the complexity of the proposal. Discussing about such things improves our understanding of what others see as good code and why: it helps us to improve our code quality  in general, even for things that are not tracked by the tools.

Wrapping up

Tools like Sonar and Jenkins really helped us to improve the quality of our code on a daily basis. Sonar is very powerful and easy to configure and the differential dashboard is really a feature that makes a huge difference. But having good tools is not enough: we need to make sure that the way we write code and the design principles we apply are consistent. Making that an open process with a formalized trace on the Wiki helped us as a team to achieve our goal.

This blog entry is not about explaining how you could define such WidgetSet and build it. The official documentation and this wiki page already explains most of it for Apache Maven. One very annoying issue with this mandatory compilation step is that it is quite slow even on modern hardware and it significantly increate the time of your build. There are several workarounds to this: reduce the number of supported browsers or make sure the result of the compilation can be cached by compiling it in the source directory of your web application project (see this blog post for a complete example).

We also tried to tackle this problem and we thought that if Vaadin can hold the result of the WidgetSet in their jar file, why shouldn’t we? And since a multi-modules build tool is a commodity these days (again, we use Maven but this should work with others too), having an extra module is not really a problem.

Our Vaadin-based projects are composed of several modules describing the different pieces of the application. For very simple project we just have one core module holding all the code that uses the Vaadin API and a webapp project (server module) to package and start the application easily. The core project might use add-ons, in which case we compile the WidgetSet in the webapp project.

Our optimization is about creating another module, depending on core and solely being responsible to compute the WidgetSet. The trick is that module is not active by default so it does not run unless you explicitly ask for it. The result of the compilation is stored in the jar file and can even be shared within the team by deploying a SNAPSHOT to a shared repository. Another nice advantage regarding this technique is that a full project clean no longer requires you to recompile the WidgetSet from your IDE since the project is or can be disabled, depending of the Maven support of your IDE

The following is an excerpt of the Maven pom we used to compile the WidgetSet in a regular jar file

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <configuration>
                <excludes>
                    <exclude>VAADIN/widgetsets/WEB-INF/**</exclude>
                </excludes>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>gwt-maven-plugin</artifactId>
            <configuration>
                <extraJvmArgs>-Xmx512M -Xss1024k</extraJvmArgs>
                <webappDirectory>${project.build.outputDirectory}/VAADIN/widgetsets</webappDirectory>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>resources</goal>
                        <goal>compile</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
        <plugin>
            <groupId>com.vaadin</groupId>
            <artifactId>vaadin-maven-plugin</artifactId>
            <executions>
                <execution>
                    <goals>
                        <goal>update-widgetset</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

The main tricks here as follows:

• We configure the GWT plugin so that the result of the compilation lands in target/classes/VAADIN/widgetsets which is the location Vaadin uses to locate a WidgetSet in the classpath
• We configure the jar plugin to remove extra things that the plugin has generated and we don’t need

Once you are done with this, you only need to add a dependency to this new module in your server module. In order to illustrate these concepts, we have created a showcase on github using the Wizards for Vaadin add-on. This showcase also illustrates a full integration with Maven as well as the use of Cargo to start the container without even the need to install one first.

And, while we were at it, we released a new version of our Embed for Vaadin add-on (see our previous blog post and the release notes).  This add-on allows you to start your Vaadin application right from your IDE. There is one class in the wizard-sample-server that you can use to see how easy this all integrate together.

As everything we do here internally, we were searching for mechanisms to optimize our daily work tasks. In the case of Vaadin, two come naturally to mind:

1. Quickly show a component while prototyping a screen or part of a screen to validate its layout, style and basic behavior
2. In case of a test case failing, be able to quickly display the faulty component to get more visual details

In Vaadin 6 (the upcoming 7 version will change that slightly), a Vaadin application needs the following servlet definition to work properly:

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
         version="2.5">

    <servlet>
        <servlet-name>vaadin</servlet-name>
        <servlet-class>com.vaadin.terminal.gwt.server.ApplicationServlet</servlet-class>
        <init-param>
            <param-name>application</param-name>
            <param-value>com.bsb.incubator.vaadin.ShowcaseApplication</param-value>
        </init-param>
    </servlet>
    <servlet-mapping>
        <servlet-name>vaadin</servlet-name>
        <url-pattern>/*</url-pattern>
    </servlet-mapping>
</web-app>

Since its version 7, Apache Tomcat has a very nice embed API that allows you to configure and start a Tomcat container without the need to package a web application (or install Tomcat, for that matter). And it offers other nice feature such as the ability to allocate a random HTTP port that is available on your box instead of forcing you to choose one.

Embed for Vaadin is a Vaadin add-on we developed that takes care of the boring tasks of configuring and starting a servlet container. For instance, the following line would start a tomcat container with an application showing a simple button

EmbedVaadin.forComponent(new Button("Hello")).start();

This code gives the following, very simple result:

Since we are embedding Tomcat, we can pass an initialized component to the application. Even though we could serialize the component and deserialize it for another user, this mode does not support multi clients and multi sessions as the purpose is for a developer to debug or prototype  a particular component.

forComponent takes any Vaadin Component. If the component is a Window, we use it as the main window of the Application; If it is a Layout, we wrap the layout in a simple Window; If it is a more basic component, we wrap it in a VerticalLayout. It is also possible to pass an Application class, just like you would do while building a regular web application. In that case, we really configure a web.xml that is similar to the one above. For instance, the following would start the ShowcaseApplication:

EmbedVaadin.forApplication(ShowcaseApplication.class).openBrowser(true).start();

Noticed the openBrowser instruction? EmbedVaadin supports more options and these can be combined very easily before starting the embed server. In this case, this instructs to open the default browser at the URL that the server is using for the application. This speeds up the process a little bit more and it works for basic component too.

By default, the thread that started the server is blocking so that you can use the web application without caring about the server lifecycle. But you could use the same mechanism for integration tests purposes. The start() instruction returns you an EmbedVaadinServer that you can stop once your test has run:

final EmbedVaadinServer server = EmbedVaadin.forComponent(myComponent).wait(false).start();
// Your test
server.stop();

To use this add-on in your application, just add the following dependency to your project:

<dependency>
    <groupId>com.bsb.common.vaadin</groupId>
    <artifactId>com.bsb.common.vaadin.embed</artifactId>
    <version>0.1</version>
</dependency>

The add-on is available on the Central repository so you should not have any problem to get it with Maven.

We believe this plugin will improve in the future. Do not hesitate to fork the repository or give us feedback!

In short, Spring 3.1 permits the use of nested <beans/> elements with a new profile attribute. If the profile defined in that attribute is enabled, the bean definitions and import statements are parsed and included in the application context. Consider for instance the following bootstrap-environment.xml configuration file:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd">

    <description>
        Bootstrap the environment according to the active profiles.
    </description>

    <beans profile="env1">
        <import resource="classpath:/META-INF/app/bootstrap/env1.xml"/>
    </beans>

    <beans profile="env2">
        <import resource="classpath:/META-INF/app/bootstrap/env2.xml"/>
    </beans>

</beans>

This basically states that if env1 is enabled, the env1.xml configuration file will be included in the application context. We could assume here that this would load something specific to that environment.

We have applications that can either deploy on a full blown JavaEE compliant server or on a lightweight container such as Apache Tomcat. A standard JavaEE server will bring a JTA transaction manager for distributed transaction and a message broker for JMS-based communication. A lightweight container, however, does not provide that infrastructure out-of-the-box so we have to bridge the gap somehow, yet keeping that maximum flexibility for customization.

Internally, we use a web application skeleton. This skeleton contains all the resources file, the web.xml and the dependencies required to build the application. We have then one project per environment were we can customize the skeleton at will, including overriding files. For that we use the overlay feature of the maven war plugin. In the end, the customization required to start the application is restricted to the environment itself.

It would be much better if we didn’t have to provide these projects in the first place and simply use a properties file containing the profile(s) to activate. This use case is not supported out-of-the-box:  Spring can determine which profile(s) to activate based on an environment property, a system property or a servlet context parameter. We do not want our users to hack their startup scripts to deploy our applications and since our web.xml is shared, we can’t really use the servlet context parameter either.

Fortunately, Spring offers an access to the lower-level API for more fine grained customization. A custom strategy simply needs to implement the ApplicationContextInitializer interface and register it properly before the context is refreshed. We have a showcase on github demoing a simple implementation of that interface looking up for a properties file at a default location (META-INF/application-context.properties). Once you have that in your project, you only need to add an extra <context-param/> to your web.xml:

<context-param>
    <param-name>contextInitializerClasses</param-name>
    <param-value>com.bsb.showcase.spring.profile.ProfilesActivationApplicationInitializer</param-value>
</context-param>

To enable a set of profiles, place a properties file at the default location in the classpath of your application:

#
# The spring profiles to activate
#
spring.profiles.active=env2

Profiles activation relies now on a simple properties file. You can add as many profiles as you want and the value can even be filtered by your build tool if you cannot hardcode the value. The advantage of this solution is that we can still use our shared infrastructure and it does not require any system or environment property.

A better stability can be achieved by limiting the amount of resources that are platform or environment dependent.

Performance, when working on projects that are several years old, and have dozens developers working on it,  can be regularly threatened by a significant growth of its codebase, making performance issues start arising in various areas.

In this article, we will cover a configuration change that will help ensuring a better build stability, along with helping in performance (effects can be different depending or current tools used).

Dealing with compilation performance problems

On some of our projects, the compilation made during the Maven build started to be significantly slower than a full build performed in Eclipse IDE by our developers .
The difference between these two executions is quite simple: Maven uses by default the javac compiler, whereas Eclipse IDE uses its own implementation, JDT Core.
With a project that has 4000+ classes to compile the difference can be huge: with a 32bit JVM having its -Xmx option set to 1280M, compilation time and memory usage are:

• 113s and 1237M for javac (1.6.0_22)
• 49s and 414M for eclipse (3.8.0.v_C03)

I hope not everyone has to deal with such a huge number of classes, but still it is a real-life sample. Smaller projects have the following build times

• ~1750 classes: 72s for javac, 11s for eclipse
• ~380 classes: 33s for javac, 5s for eclipse

One interesting thing about javac is its memory footprint, which is always way higher than eclipse. In Maven projects that have tons of modules, that can lead to an additional performance gain.

So, Maven has be configured to use JDT compiler. This should be straightforward as stated in the compiler plugin page:

<plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>2.3.2</version>
    <configuration>
        <compilerId>eclipse</compilerId>
    </configuration>
    <dependencies>
        <dependency>
            <groupId>org.codehaus.plexus</groupId>
            <artifactId>plexus-compiler-eclipse</artifactId>
            <version>1.8.1</version>
        </dependency>
    </dependencies>
</plugin>

This works most of the time, but for some particular cases, it is not good enough (case sensitivity issues, debugging problems, log output to be improved), and as we really needed something to help compilation time, we started working on adapting the plexus-compiler-eclipse to be usable for all our projects (more than 2 million LOC!). The patch containing everything needed is attached to the Codehaus PLXCOMP-173 issue. I guess that most of projects can be compiled without this, but some old ones should need it.

Using the eclipse implementation, the compiler plugin gets all it needs from Maven dependencies , reducing potential build instability.

No issue has been found regarding the potential bytecode difference.

Unfortunately, this issue we raised is still unassigned, and seeing the current project activity, will unlikely be resolved in a near future. As we did, it is nevertheless possible to create your own artefact by patching the plexus-compiler-eclipse 1.8.1 source code and installing it to your own repository.

Annotation processing performance

Annotation processing is also something that can have a significant performance issue during Maven builds.

The processing can be made in two ways:

• Using one of the specific plugins (maven-processor-plugin, apt-maven-plugin)
• Performing processing during compilation (introduced with JSR-269, supported since maven-compiler-plugin 2.2)

The first option sometimes led us to processing times longer that 1 minute. This seemed to be dependent of the number of classes for the project, but also the number of its dependencies. For one of our projects, annotation processing was taking more time that all other phases together. The plugins will in the end either call javac of apt executables.

We couldn’t manage to perform annotation processing with plexus-compiler-eclipse, probably because of the eclipse compiler implementation chosen by the component. Going back to use javac as compiler in Maven builds was not an option, and would not have improved APT time.

Fortunately, there was another compiler component that was using JDT, and was close to our requirement regarding APT: tycho-compiler-jdt, a component of the Tycho Eclipse project.

Configuring  tycho-jdt-compiler as the one  used within Maven builds is quite easy:

<plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>2.3.2</version>
    <configuration>
        <compilerId>jdt</compilerId>
    </configuration>
    <dependencies>
        <dependency>
            <groupId>org.eclipse.tycho<groupId>
            <artifactId>tycho-compiler-jdt</artifactId>
            <version>0.13.0</version>
        </dependency>
    </dependencies>
</plugin>

This configuration is good enough for compiling Java classes with an up-to-date version of the JDT compiler. And is also available directly from Maven Central, so no patching and custom packaging  has to be done.

But that component is just missing a few enhancements to be able to process annotations during compilation. As for the plexus-compiler-eclipse, we made these and proposed the patch to the project community (see TYCHO-590 and Eclipse bugzilla bug #360427). That way, we can use annotation processing within the maven-compiler-plugin, whatever compilerId is chosen.

Annotation processing time on the most problematic module fell from 80 seconds to 5 seconds, although most of the time the change was from 20 seconds to 2 seconds for one module.

As for the compiler, using jdt implementation means that nothing related to the running JVM or JAVA_HOME is potentially affecting the processing output.

Impact on stability

The advantage about using Eclipse JDT for Java compilation and annotation processing is that your compilation phase doesn’t rely on any system configuration (here, mainly JAVA_HOME environment variable).  Your compilation and annotation processing stay identical whatever JVM is used. Although javac regularly improves its compilation performance,  being sure that the compilation output stays identical on any potential environment is a significant plus.

Other paths for improving compilation and annotation processing

• Recent Oracle JDKs (1.7 and 1.6.0_25+) provide significant improvements regarding compilation time, memory footprint stays huge.
• Oracle JRockit 1.6 has APT performance close to Eclipse JDT. Compilation time is also a lot better than pre Oracle 1.6.0_25 JDKs, but it also uses a lot of memory.

Spring Batch provides a default PartitionHandler implementation that delegates the processing to a TaskExecutor (see TaskExecutorPartitionHandler). What it basically does is wrapping the execution of the partition in a FutureTask and delegating its processing to the underlying TaskExecutor (a thread pool). Once each FutureTask has completed, the implementation simply returns the StepExecution instances that each task has returned. Simple !

Our approach is to take those simple principles and see how they can be applied to a remote execution where JMS would be used to transport the requests and responses. Executing the partition remotely brings new challenges:

1. The remote worker does not have the actual Step instance so it needs to be able to retrieve it based on a reference
2. Compared to the default PartitionHandler, we need to wait for responses coming from different JVM(s)

Retrieving the Step instance

In order to provide a complete isolation for each batch, Spring Batch provides the ability to declare each job in a dedicated child application context. When the application starts with this feature, a component scans all the jobs matching a configurable expression and creates a child context for each of those files (we specify one job per file). For instance, the following configuration scans (and therefore register) all jobs declared in a XML files found in the config root package and starting with ‘job’.

<bean class="org.springframework.batch.core.configuration.support.AutomaticJobRegistrar">
    <property name="applicationContextFactories">
        <bean class="org.springframework.batch.core.configuration.support.ClasspathXmlApplicationContextsFactoryBean">
            <property name="resources" value="classpath*:/config/job*.xml"/>
        </bean>
    </property>
    <property name="jobLoader">
        <bean class="org.springframework.batch.core.configuration.support.DefaultJobLoader">
            <property name="jobRegistry" ref="jobRegistry"/>
        </bean>
    </property>
</bean>

The  JobLoader updates a component called the JobRegistry which is able to locate a Job by its name. This is the piece we have updated by introducing the concept of StepRegistry:

/**
 * Registry keeping track of all the {@link Step} defined in a
 * {@link org.springframework.batch.core.Job}.
 */
public interface StepRegistry {

    /**
     * Registers all the step of the given job. If the job is already registered,
     * the method {@link #unregisterStepsFromJob(String)} is called before registering
     * the given steps.
     *
     * @param jobName the give job name
     * @param steps the job steps
     */
    void register(String jobName, Collection<Step> steps);

    /**
     * Unregisters all the steps of the given job. If the job is not registered,
     * nothing happens.
     *
     * @param jobName the given job name
     */
    void unregisterStepsFromJob(String jobName);

    /**
     * Returns the {@link Step} of the specified job based on its name.
     *
     * @param jobName the name of the job
     * @param stepName the name of the step to retrieve
     * @return the step with the given name belonging to the mentioned job
     * @throws NoSuchJobException no such job with that name exists
     * @throws NoSuchStepException no such step with that name for that job exists
     */
    Step getStep(String jobName, String stepName) throws NoSuchJobException, NoSuchStepException;

}

And of course, our JobLoader now updates the content of this registry.

Aggregating the result

Our partition handler needs now to send one StepExecutionRequest per partition to run, as a JMS message. Let’s ignore for now what happens on the remote end; we still need to deal with the results aggregation. In Spring Batch, one thread (the master) is managing the state of the entire job. So this thread must wait (i.e. block) until all responses have been received from the workers. In a cluster, we never know where the master thread will be so we had to create a mechanism where each job has a temporary queue associated to it. Having a temporary queue makes sure that only the master thread can listen for messages. And because we set the reference to this temporary queue as the JMSReplyTO header of each request, our remote workers are able to send the response to that queue as well.

Remains a classical problem in asynchronous communication: timeout. To deal with this problem, we have built a JmsAggregationService that is managed by the following interfaces:

• AggregationItemListener: receives an event when a response is received by the aggregation service.
• AggregationCompletionPolicy: determine when the aggregation is done, regardless of a potential timeout. In our case, this policy is straighforward since we expect one response per request so the only thing we have to do in this case is to use a default implementation that counts the number of responses
• AggregationTimeoutPolicy: determines if a timeout should occur based on the time at which the aggregation started

While we could have built a straightforward policy that would timeout after some time, we decided to build an implementation that benefits from the metadata that Spring Batch stores in its internal model. Every time a chunk completes, the thread contributes some metadata to its StepExecutionContext and flushes it to the database. This is very handy because we can easily find out if a given partition is running or the last time it contributed something.

Based on these facilities, the aggregation service performs the following:

1. Creates a JMS session
2. Creates a temporary JMS queue that will be used for the aggregation
3. Sends one message per request. Each message holds a reference to the temporary JMS queue
4. Listen for incoming response
   • When no response has been received, it ask the timeout policy if it should timeout. Based on the result, it either throws an exception or listen again for a response
   • When a response has been received, it invokes all registered listeners and ask if it should complete. Based on the result, it either returns the result or listen again for a new response

Executing the partition on the remote worker

Executing the actual partition is now easy. Upon the reception of a StepExecutionRequest, the worker retrieves the actual Step instance using the StepRegistry. Once the partition completes or if an error occurs a response is sent to the destination that was set in the JMSReplyTO header of the request.

The diagram below summarizes the global solution

Note that even though the master seems to be separate from the actual slaves, it does not have to be that way in practice. This video demonstrates how an additional node can join a running instance. It also showcases what happens when a node is killed in the middle of a job!

Click here to view the embedded video.

Conclusion

These developments allows to support a remote partitioning use case for Spring Batch. And the good news is that BSB open sourced them (see the spring-batch and spring-batch-admin forks on github!). It should be noted that not a lot of code is actually JMS specific: most of the infrastructure can be reused to implement use cases where the transport protocol is different. Only the aggregation service and the remote invoker should be changed.

1. We have multiples ways to handle an error. Some issues might force the batch execution to fail. Others could be ignored and treated specifically by a next attempt
2. We need to be able to restart where we left-off
3. Processing should be distributed (partitioning)

Therefore such a job starts with a first staging step whose purpose is to execute a query on the domain model and feed the staging table with the items to process. Each row in this staging table represents a staging item and has the following information:

• a technical identifier (primary key)
• a reference to the job instance (jobInstanceId)
• a sequence number: the sequence starts at 0 and is incremented for every new item
• the name of the last step that handled the item
• the current status of the item which could be either PENDING, COMPLETED, ERROR or SKIPPED
• the actual item

The staging step is a regular, chunk-oriented, Spring Batch step. It has an ItemReader, an ItemProcessor and a no-op ItemWriter. The item reader can be any item reader, really. For the explanation, say that our reader is reading items from a database. Our item processor might look like the following:

/**
 * A default {@link StagingItemProcessor} that returns a {@link StagingItemProvider}
 * instance with no custom staging information by default. Sub-classes can override
 * the {@link #getStagingValue(Object)} or {@link #getCustomStagingItemProvider(Object)}
 * methods to override the default behavior.
 */
public class DefaultStagingItemProcessor<I> implements StagingItemProcessor<I> {

    private StagingValueProvider<I> stagingValueProvider;

    public StagingItemProvider process(I item) {
        return new StagingItemProvider(getStagingValue(item),
                getCustomStagingItemProvider(item));
    }

    /**
     * Returns the {@link StagingValue} of the specified item.
     *
     * @param item the item
     * @return the staging value
     */
    protected StagingValue getStagingValue(I item) {
        return stagingValueProvider.getStagingValue(item);
    }

    protected CustomStagingItemProvider getCustomStagingItemProvider(I item) {
        return null;
    }

    /**
     * Sets the {@link StagingValueProvider} to use.
     *
     * @param stagingValueProvider the staging value provider
     */
    public void setStagingValueProvider(StagingValueProvider<I> stagingValueProvider) {
        this.stagingValueProvider = stagingValueProvider;
    }
}

What this code shows is that the staging item processor is responsible to process an incoming item and return a StagingItemProvider instance based on a StagingValueProvider.  The StagingValueProvider is responsible to provide the staging value out of a given item, this value being the item field in the staging table. Say for instance that a set of policies have been read from the database, an instance of StagingValueProvider would return a StagingValue holding the primary key of the policy. We use a typed object (StagingValue) and not the raw Serializable item because it contains additional metadata such as how to actually store it. Indeed, the staging table has three columns to hold the item, a LONG, a VARCHAR and a CLOB depending on the serialization strategy that was chosen. The CustomStagingItemProvider, which is null in this case is used for business chunking that we will cover later.

The framework uses an internal ItemProcessor delegating to this one to actually create the staging items in the database. When the transaction commits for the chunk, one staging item per item to process is added to the staging table (which explains why the writer is a no-op).

Once the staging step has completed, our staging item table is filled for a particular job instance and the actual processing can start. A step that relies on this mechanism is called a staged step: it will read the staging value and process the actual item. Before processing the actual item, the staging value needs to be translated back to the actual item. This is done through an implementation of the StagingValueReader. In the previous example, such instance might retrieve the actual policy out of the database based on the primary key. In practice, our developers have to pick the instance to use based on the ones we provide for most of our use cases.

Since every item in the job has a status and a step name, it is possible to skip an item and move along with the rest of the data to process. In this case, the job fails with a COMPLETED WITH ERRORS status. Restarting the job would restart the processing of failed items at the step mentioned in the staging table, giving the ability to restart exactly where we left-off.

Partitioning

Moreover, the sequence number allows to easily identify the boundaries of partitions, allowing to distribute the items to process to several workers. Our Partitioner simply puts the min and max sequence for each partition in the execution context. Our reader gets these information from the execution context and performs an SQL query that looks like the following (simplified):

SELECT OBJECTID, STATUS, SEQUENCE
FROM STAGING_ITEM
WHERE JOBINSTANCEID = 1234 AND SEQUENCE >= 0 AND SEQUENCE <= 999 AND (STATUS = 'PENDING' AND STEPNAME = 'step1')
ORDER BY SEQUENCE;

where 1234 is the id of the job instance, step1 is the name of the staged step that is running and dealing with a partition of 1000 elements (in this case the first partition). Using this procedure, each partition reads the data it has to process independently (and also updates the status of each individual item independently).

Business chunking

Simple partitioning might not be enough in more complex use cases. Often the data to process (for instance policies) is linked to another business concept (for instance the customer) and it is important to process all the items related to that concept in the same thread. One obvious reason is data aggregation that should be stored somewhere when all the policies for a particular customer have been handled. To achieve this scenario, each partition must contain a set of business chunks and we need to be able to identify them. The way we have chosen to fulfill this need is to open the possibility for developers to stores arbitrary data in a custom table that is linked to the staging item with a foreign key.

Back on our example, a batch needs to process all the policies of the system and we want to perform some aggregation on the customer.

• Item type to process: the policy
• Business chunk: all the policies of a customer

During the staging process, we store additional information in a table called policy_custom_item. The developer can provide an implementation of the CustomStagingItemProvider to store the custom data at the same time as the default one. Again, we provide a default implementation, based on conventions, where the developer has to set the name of the custom table and a map of column names to values and the framework will deal with the primary key of the row and the reference to the item in the staging_item table (column stagingitemid).

We should then give the list of business partitions (one business partition here means all the policies of a given customer), something like:

SELECT MIN(S.sequence) as minSeq, MAX(S.sequence) as maxSeq
FROM staging_item S, policy_custom_item C
WHERE S.jobinstanceId = ? AND C.stagingitemid = S.objectid
GROUP BY C.clientid ORDER BY maxSeq

Our custom Partitioner is now responsible to rebuild the partitions based on these business chunks. In this case, it means that the size of the partitions might differ, especially if the number of policies per customer differ a lot. Notice that the only thing that the developer had to do in this case is provide the SQL query used to build the business chunk boundaries!

The processor of such a step is also able to detect that the business chunk has changed and offer the ability to implement callbacks offered  by the BusinessChunkListener interface (beforeBusinessChunk, afterBusinessChunk).

Conclusion

Spring Batch offers an open contract that allowed us to build an infrastructure that suited our needs. There is no demo available at this time as the features that we have exposed in this article are part of other features that we have built on top of Spring Batch. In a next article we will cover how partitions can be invoked on several machines, offering a distributed partitioning support for Spring Batch.