Comparison of Continuous Integration Servers

A Short Background

Continuous integration is a practice the grew out of the extreme programming community, and was written about by Martin Fowler and Kent Beck. Martin Fowler’s paper provides a very nice definition of the term

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. This article is a quick overview of Continuous Integration summarizing the technique and its current usage.

When being used effectivly, the practice of continious integration provides feedback on the ‘health’ of the software after every commit to the software repository allowing developers to identify and fix defects early. Since the time between the commit and the identification of a problem is so short, bugs are often simpiler as eassier to fix.

Laying the out the guidelines

There are many continuous integration servers out there and they each have their own pros and cons, but just to shrink the playing field a little bit I’m going to layout some basic criteria. So I am looking for the following in a continuous server:

  • Free, I know that there are numerous COTS solutions, but free should fit into everyone’s budget. Also with as many good free solutions available with you really don’t need a COTS product.
  • Should work using existing projects build files
  • Should work with a popular SCM system, I’m looking for at least SVN, CVS
  • Should have an adequate selection of notification systems

I know that these are pretty vague but these criteria do eliminate many solutions from the playing field, and in my opinion the solutions that are left are going to be the more mainstream and stable solutions.

So here we go, in no particular order:

CruiseControl

CruiseControl is a product that I’ve had a love/hate relationship with for many years, while it does its job and possesses all the fundamental qualities that you would look for in an it is rather cumbersome to configure, and not the most refined thing to look at. That being said it is a work horse and it does get the job done.

Installation

The installation of cruisecontrol is pretty straight forward and about on par for what you might expect for any opensource application. The quickstart from their website reads as follows:

Unzip the release to a directory, like <em>~/cc-sanbox/cruisecontrol-bin-2.7.3</em>

<span style="font-family: 'Lucida Grande';"><em>Check that the scripts </em><code><em>cruisecontrol-bin-2.7.3/cruisecontrol.sh</em> and <em>cruisecontrol-bin-2.7.3/apache-ant-1.7.0/bin/ant</em> have execution permission.</span></code>

<span style="font-family: 'Lucida Grande';"><em>Run </em><code><em>~/cc-sanbox/cruisecontrol-bin-2.7.3/cruisecontrol.sh</em> from the newly unzipped directory</span></code>

In all honesty that is a pretty accurate assessment of the level of effort it takes to install the software and start the application.

Configuration

All the configuration for CruiseControl is done in with XML files, and there is no good utility for helping you setup a project other than the configuration reference, a ton of elbow grease, and maybe a few cold ones. I think that if you take a look at the configuration reference, it is pretty daunting at first glance, and second glance … right up to about the 100th or so. It is just a lot of information to take in, and while it is very well documented and makes perfect sense once you get the hang of it, it will take a bit of time and experimentation until you’ve hit that balance of functionality that you were hoping to achieve from the system.

User Interface

There are actually two user interfaces that come with CruiseControl.

The standard interface is kinda of a web 1.0 older looking interface, although it is powered off of XSLT stylesheets, and serverl other new(er) technologies, it just doesn’t bring in the easy to use kind of appeal to the product.

This is the main status page, it is a fairly simple overview of all the projects that the server is tracking, the status of the build and the last time a build was run. </p>

It provides a link into the project so you can investigate any failures or access any artifacts produced by that build.</td> </tr>

This screen gives you a detailed view of a particular project. On the left you can pick which build you want to look at so you can compare the current build to a past build. </p>

This view is customizable if you have the stomach for editing the stylesheets, and you can add support for the output of any other build tool that can produce xml output like findbugs, checkstyle, pmd etc..</td> </tr> </tbody> </table>

The dashboard interface is a bit better, it uses AJAX and provides a more modern feel.

Like above this view gives you an overall sense of how your projects are doing, this view with the color-coded boxes allows you to keep an eye on the status of many projects and mousing over any box will give you more information like project name and last build time.
Much like the screen in the old user interface, this allows you to see information like which tests are failing, what the changeset was that triggered the build. And you can navigate to previous builds.

While I do consider the dashboard interface an improvement over the standard interface, it still lacks a polished quality. But that aside it does allow you to do everything and anything you could want to do with it, you just might have to go into the source and do some tweeking.

Overall

While CruiseControl probably does everything that you would like it to do, its a bit hard on the eyes, and I talked with many users of it and they have all expressed the same feelings of mild irritation when it comes to configuring the server. All that being said it is the original so it gets points for that.

LuntBuild

Installation

Installation of LuntBuild is pretty straight forward, it has the option of an installation GUI.

Download Luntbuild installer from Luntbuild Sourceforge site, or from Luntbuid Javaforge  site
Run command java -jar luntbuild-xxx-installer.jar
cd <luntbuild-install-dir>
java -jar luntbuild-standalone.jar localhost 8888
</luntbuild-install-dir>

Now if you don’t mind it using teh HSQL DB embedded database, then you’re done, if you want it to use another type of database then there are instructions to set that feature up here.

Configuration

All the LuntBuild configuration is done within the web interface, select the projects tab and click the (new project) icon.

You can then start customizing the fields to your hearts desire. However most all of the fields are just text input boxes and its up to you to select the correct thing.

User Interface

The main status view shows you the status of each project’s last build, this view also provides links into the project configuration and into the details of the last build.
The build status view, shows you all the basic information about the status of the build. Not to many bells and whistles here just a basic view.

Overall

LuntBuild seems that it will get the job done, but it isn’t going to win any awards doing it, it seems to be a product that is just kind of middle of the road ‘eh’

Continuum

Continuum is a lightweight brought to you by the team who developed Maven. It is has a fairly simple installation and a novice amount initial configuration.

Installation

The installation for continuum is pretty straight forward

  • Download the standalone version from the Download page
  • Extract the file
  • Set a JAVA_HOME environment variable which use a jdk >= 1.5
  • configure JNDI resources (such as SMTP server, database, etc..)

It is actually this last step that seems the most complicated, because it involves mucking around in the jetty XML files to define the resources.

Configuration

Screen that allows you to configure a new Ant project
Screen that allows you to edit a project

User Interface

For my tastes the Continuum interface is a little daunting looking, it is defiantly utilitarian. But I don’t know that it is particularly user friendly. I prefer interfaces that use constrained input fields that are harder for new users to screw up.

Overall

While Continuum is a full featured server, I’m not sure that it would be very well used, the harsh nature of the user interface doesn’t really help users want to use it. I believe that Continuum would likely be deployed and then quickly fall into disrepair.

Hudson

Installation

You will never find a product that is as easy to install as Hudson. Just download the file and run

java -jar hudson.war

You will be off and running, no messing around in configuration files or anything. Now if you really want to mess with the configuration files (and you really don’t have to, because everything can be done in the UI) they are available and you can hack away to your hearts content.

Configuration

This is just an example of one of the many screen you find to help you configure parts of the system, everything is in small manageable pieces and almost all (if not all) have examples and context help to show you what goes where,
This is a view of available plugins that you can select and just press install and hudson will go out and upgrade itself, you can’t ask for more than that.

User Interface

This is the main screen, it is simple and elegant and it tells you everything you want to know about all the projects as well as what the server is working on building now, and what is in the build queue.
This is a typical screen that displays the project summary, you can see that the it is giving you the trend for unit test failures in the form of a graph which I love. Most of the project detail screen are similar to this one, favoring pictures over text so you can get all the information you need at a quick glance.

Overall

I love this product, installation is a snap and configuration is extremely simple but the most powerful thing about hudson is its plugin API it has allowed many developers (including your’s truly) to contribute to the project creating an impressive wealth of functionality.

And I don’t think that I’ve found a single thing that the other projects can do that hudson doesn’t do as well if not better.

Conclusions

Of course it depends on what you are looking for, cruisecontrol is going to be a bit more work up front but once you get it going, it will serve you well as long as you don’t mind its old school user interface. Whereas hudson will be up and running within minutes, and it boasts a very modern very easy user interface.

I’m not sure that I would waste my time with either LuntBuild or Continuum, they are both solid projects but neither presents a very compelling reason to choose them over CrusieControl or Hudson.

Since I believe that user interface is a critical piece of a continuous integration server I’m gonna give this one to Hudson. Since basically a CI server is something who’s whole purpose for existence is to nag you when you’ve done something wrong, and then help you narrow down what you did and when you did it. If you don’t have a descent interface then you developers will slowly start to ignore its feedback.