Comparison of Continuous Integration Servers26 Sep 2008
A Short Background
Continuous integration is a practice the grew out of the extreme programming community, and was written about by Martin Fowler and Kent Beck. Martin Fowler’s paper provides a very nice definition of the term
Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. This article is a quick overview of Continuous Integration summarizing the technique and its current usage.
When being used effectivly, the practice of continious integration provides feedback on the ‘health’ of the software after every commit to the software repository allowing developers to identify and fix defects early. Since the time between the commit and the identification of a problem is so short, bugs are often simpiler as eassier to fix.
Laying the out the guidelines
There are many continuous integration servers out there and they each have their own pros and cons, but just to shrink the playing field a little bit I’m going to layout some basic criteria. So I am looking for the following in a continuous server:
- Free, I know that there are numerous COTS solutions, but free should fit into everyone’s budget. Also with as many good free solutions available with you really don’t need a COTS product.
- Should work using existing projects build files
- Should work with a popular SCM system, I’m looking for at least SVN, CVS
- Should have an adequate selection of notification systems
I know that these are pretty vague but these criteria do eliminate many solutions from the playing field, and in my opinion the solutions that are left are going to be the more mainstream and stable solutions.
So here we go, in no particular order:
CruiseControl is a product that I’ve had a love/hate relationship with for many years, while it does its job and possesses all the fundamental qualities that you would look for in an it is rather cumbersome to configure, and not the most refined thing to look at. That being said it is a work horse and it does get the job done.
The installation of cruisecontrol is pretty straight forward and about on par for what you might expect for any opensource application. The quickstart from their website reads as follows:
Unzip the release to a directory, like
<span style="font-family: 'Lucida Grande';"><em>Check that the scripts </em><code><em>cruisecontrol-bin-2.7.3/cruisecontrol.sh</em> and
<em>cruisecontrol-bin-2.7.3/apache-ant-1.7.0/bin/ant</em> have execution permission.</span></code>
<span style="font-family: 'Lucida Grande';"><em>Run </em><code><em>~/cc-sanbox/cruisecontrol-bin-2.7.3/cruisecontrol.sh</em> from the newly unzipped directory</span></code>
In all honesty that is a pretty accurate assessment of the level of effort it takes to install the software and start the application.
All the configuration for CruiseControl is done in with XML files, and there is no good utility for helping you setup a project other than the configuration reference, a ton of elbow grease, and maybe a few cold ones. I think that if you take a look at the configuration reference, it is pretty daunting at first glance, and second glance … right up to about the 100th or so. It is just a lot of information to take in, and while it is very well documented and makes perfect sense once you get the hang of it, it will take a bit of time and experimentation until you’ve hit that balance of functionality that you were hoping to achieve from the system.
There are actually two user interfaces that come with CruiseControl.
The standard interface is kinda of a web 1.0 older looking interface, although it is powered off of XSLT stylesheets, and serverl other new(er) technologies, it just doesn’t bring in the easy to use kind of appeal to the product.
This is the main status page, it is a fairly simple overview of all the projects that the server is tracking, the status of the build and the last time a build was run. </p>
It provides a link into the project so you can investigate any failures or access any artifacts produced by that build.</td> </tr>
This screen gives you a detailed view of a particular project. On the left you can pick which build you want to look at so you can compare the current build to a past build. </p>
This view is customizable if you have the stomach for editing the stylesheets, and you can add support for the output of any other build tool that can produce xml output like findbugs, checkstyle, pmd etc..</td> </tr> </tbody> </table>
The dashboard interface is a bit better, it uses AJAX and provides a more modern feel.
While I do consider the dashboard interface an improvement over the standard interface, it still lacks a polished quality. But that aside it does allow you to do everything and anything you could want to do with it, you just might have to go into the source and do some tweeking.
While CruiseControl probably does everything that you would like it to do, its a bit hard on the eyes, and I talked with many users of it and they have all expressed the same feelings of mild irritation when it comes to configuring the server. All that being said it is the original so it gets points for that.
Installation of LuntBuild is pretty straight forward, it has the option of an installation GUI.
Download Luntbuild installer from Luntbuild Sourceforge site, or from Luntbuid Javaforge site Run command java -jar luntbuild-xxx-installer.jar cd <luntbuild-install-dir> java -jar luntbuild-standalone.jar localhost 8888 </luntbuild-install-dir>
Now if you don’t mind it using teh HSQL DB embedded database, then you’re done, if you want it to use another type of database then there are instructions to set that feature up here.
You can then start customizing the fields to your hearts desire. However most all of the fields are just text input boxes and its up to you to select the correct thing.
LuntBuild seems that it will get the job done, but it isn’t going to win any awards doing it, it seems to be a product that is just kind of middle of the road ‘eh’
Continuum is a lightweight brought to you by the team who developed Maven. It is has a fairly simple installation and a novice amount initial configuration.
The installation for continuum is pretty straight forward
It is actually this last step that seems the most complicated, because it involves mucking around in the jetty XML files to define the resources.
For my tastes the Continuum interface is a little daunting looking, it is defiantly utilitarian. But I don’t know that it is particularly user friendly. I prefer interfaces that use constrained input fields that are harder for new users to screw up.
While Continuum is a full featured server, I’m not sure that it would be very well used, the harsh nature of the user interface doesn’t really help users want to use it. I believe that Continuum would likely be deployed and then quickly fall into disrepair.
You will never find a product that is as easy to install as Hudson. Just download the file and run
You will be off and running, no messing around in configuration files or anything. Now if you really want to mess with the configuration files (and you really don’t have to, because everything can be done in the UI) they are available and you can hack away to your hearts content.
I love this product, installation is a snap and configuration is extremely simple but the most powerful thing about hudson is its plugin API it has allowed many developers (including your’s truly) to contribute to the project creating an impressive wealth of functionality.
And I don’t think that I’ve found a single thing that the other projects can do that hudson doesn’t do as well if not better.
Of course it depends on what you are looking for, cruisecontrol is going to be a bit more work up front but once you get it going, it will serve you well as long as you don’t mind its old school user interface. Whereas hudson will be up and running within minutes, and it boasts a very modern very easy user interface.
I’m not sure that I would waste my time with either LuntBuild or Continuum, they are both solid projects but neither presents a very compelling reason to choose them over CrusieControl or Hudson.
Since I believe that user interface is a critical piece of a continuous integration server I’m gonna give this one to Hudson. Since basically a CI server is something who’s whole purpose for existence is to nag you when you’ve done something wrong, and then help you narrow down what you did and when you did it. If you don’t have a descent interface then you developers will slowly start to ignore its feedback.