Maven | DevOpsNet

Maven the Version Number Nazi

Posted on May 4, 2012 by jamesbetteley

Maven doesn’t like it when you use different verison numbers to the Maven standard format. Of course it doesn’t. It wouldn’t would it? It’s Maven, and Maven only likes it when you do what it tells you to do. I’m still a bit annoyed with Maven, as you can probably tell.

Don’t get me wrong, I’m not “Maven bashing”, it’s just that this particular problem doesn’t have quite the elegant solution I was looking for. I do appreciate Maven, honestly.

This was the problem:

I wanted to change our versioning system from something like 1.0.0-1234 to something like 1.0.0-1234-01

Why the hell would I want to do that?? I’ll explain…

Our verisoning is like this:

{major}.{minor}.{patch}-{build}

The only problem was, the build number was taken from the Perforce check-in number, and this number didn’t always change whenever a build was made, especially if the build was kicked off by an upstream dependency, or a forced build was triggered. Basically, if the build was kicked off by anything other than a commit to Perforce, the build would create an artifact of identical version to the previous build. This, in theory, shouldn’t be a problem, because it is actually building exactly the same thing, but I just don’t like it. Anything could happen, any environmental change could produce a slightly different build to the previous one.

The problem was that I wasn’t using an incremental counter anywhere in my version numbe. It’s essential to have an incrementing version number in order to ensure that every single build creates a unique identifier, so that no two different builds can appear to be the same build.

My first thought was to append a build counter on the end, like this:

{major}.{minor}.{patch}-{build}-{counter}

And that would have worked fine, if it wasn’t for the fact that we use version ranges in our dependencies, and we already have plenty of builds which use the previous versioning system. Maven kept picking up the builds with the previous version system, even though, in every possible sense, the new ones had higher version numbers. It made no sense. That’s when I looked into how maven works out versions. Basically it says “if you’re using version ranges, and not using the maven standard versioning format, you might as well forget it”. If it sees dependencies using the standard format, and ones using the non-standard maven format, it’ll pick up the standard format ones and basically ignore the new ones. To get around this you can delete all the old builds using the standard maven format, and then it’ll work, because it’ll treat each build version like a string and just get you the latest in whatever your range is.

Sadly, this isn’t an option for me, as I want to kep the old builds using the old format. So I tried a few things. I tried putting a string in as a separator, so it would look like this:

{major}.{minor}.{patch}-{build}rc{counter}

This effectively produces something looking like:

1.0.0-1234rc01

I’m fine with that. Maven, on the other hand, isn’t. I made a build with this version 1.0.0-9999rc01 and used it as a dependency in another build, but the other build still went and got 1.0.0-1234, the OLD build using the standard maven versioning. I mean, you’d think 1.0.0-9999rc01 > 1.0.0-1234 but apparently not.

I was a bit pushed for time so I couldn’t spend forever looking into this, so I’ve basically just appended the build counter directly onto the end of the perforce number. This works ok, but just looks a little ugly.

There’s more information on the Maven versioning rules here. It seems that you can break the rules no problems, but you’re in trouble if you use version ranges in your dependencies, and your dependencies need to live alongside binaries which use the standard maven versioning system 😦

If anyone has any better solutions I’d like to hear them. And please don’t say “stop using Maven”.

Exporting MAVEN_OPTS not working for Freestyle Jenkins Projects

Posted on November 10, 2011 by jamesbetteley

I’m running Jenkins version 1.428 and I have a “Freestyle” project. The aim of this project is to upload a large tar.gz file to an artifactory repository. The team are using Maven so we started out by looking at using Maven to do the deployment. It seems simple enough – you just choose “Invoke top-level maven targets” and add the usual maven deploy details here, like so:

The trouble is, because the tar is so large (126Mb in this instance), it fails with a Java heap space issues (out of memory exception). So, it would be very tempting to try to increase the MAVEN_OPTS by passing it via the JVM Options, as shown below:

Sadly, this doesn’t work. It fails with an ugly error saying MAVEN_OPTS isn’t recognised, or some such thing.

Now, in an attempt to be clever, we thought we could simply execute a shell command first, and get that to set the MAVEN_OPTS, like so:

Good ideah huh? Yeah, well it didn’t work 😦

It looks like there’s a general issue with setting MAVEN_OPTS for freestyle projects in Jenkins version 1.428 – see here for details.

Workarounds:

There’s actually a couple of workarounds. Firstly, you can run maven via the shell execution command, and explicitly pass the MAVEN_OPTS value:

This actually works fine, and is the preferred way of doing it in our case.

The other option is to use Ant to do the deployment – this somehow seems to use a lot less memory and doesn’t fail quite so easily. Also, you don’t have this issue with the MAVEN_OPTS being passed as you can set it directly in the ant file. Here’s the ant file I created (I also created a very basic pom.xml):

<project name=”artifactory_deploy” basedir=”.” default=”deploy_to_repo” xmlns:artifact=”antlib:org.apache.maven.artifact.ant”>
<description>Sample deploy script</description>
<taskdef resource=”net/sf/antcontrib/antcontrib.properties”/>

<property name=”to.name” value=”myrepo” />
<property name=”artifactory.url” value=”http://artifactory.mycompany.com”/>
<property name=”jar.name” value=”massive.tar.gz”/>

<target name=”deploy_to_repo” description=”Deploy build to repo in Artifactory” >
<artifact:pom id=”mypom” file=”pom.xml” />
<artifact:deploy file=”${jar.name}”>
<remoteRepository url=”${artifactory.url}/${to.name}” />
<pom refid=”mypom” />
</artifact:deploy>
</target>

</project>

Coping With Big C.I.

Posted on October 20, 2011 by jamesbetteley

Last night I went along to another C.I. meetup to listen to Tom Duckering, a consultant devops at Thoughtworks, deliver a talk about managing a scaled-up build/release/CI system. In his talk, Tom discussed Continuous Delivery, common mistakes, best practices, monkeys, Jamie Oliver and McDonald’s.

Big CI and Build Monkeys

First of all, Tom started out by defining what he meant by “Big CI”.

Big CI means large-scale build and Continuous Integration systems. We’re talking about maybe 100+ bits of software being built, and doing C.I. properly. In Big CI there’s usually a dedicated build team, which in itself raises a few issues which are covered a bit later. The general formula for getting to Big CI, as far as build engineers (henceforth termed “build monkeys”) are concerned goes as follows:

build monkey + projects = build monkeys

build monkeys + projects + projects = build monkey society

build monkey society + projects = über build monkey

über build monkey + build monkey society + projects = BIG CI

What are the Issues with Big CI?

Big CI isn’t without its problems, and Tom presented a number of Anti-Patterns which he has witnessed in practice. I’ve listed most of them and added my own thoughts where necessary:

Anti-Pattern: Slavish Standardisation

As build monkeys we all strive for a decent degree of standardisation – it makes our working lives so much easier! The fewer systems, technologies and languages we have to support the easier, it’s like macro configuration management in a way – the less variation the better. However, Tom highlighted that mass standardisation is the work of the devil, and by the devil of course, I mean McDonald’s.

McDonald’s vs Jamie Oliver

Jamie Oliver might me a smug mockney git who loves the sound of his own voice BUT he does know how to make tasty food, apparently (I don’t know, he’s never cooked for me before). McDonald’s make incredibly tasty food if you’re a teenager or unemployed, but beyond that, they’re pretty lame. However, they do have consistency on their side. Go into a McDonald’s in London and have a cheeseburger – it’s likeley to taste exactly the same as a cheeseburger from a McDonald’s in Moscow (i.e. bland and rubbery, and nothing like in the pictures either). This is thanks to mass standardisation.

Jamie Oliver, or so Tom Duckering says (I’m staying well out of this) is less consistent. His meals may be of a higher standard, but they’re likely to be slightly different each time. Let’s just say that Jamie Oliver’s dishes are more “unique”.

Anyway, back to the Continuous Integration stuff! In Big CI, you can be tempted by mass standardisation, but all you’ll achieve is mediocrity. With less flexibility you’re preventing project teams from achieving their potential, by stifling their creativity and individuality. So, as Tom says, when we look at our C.I. system we have to ask ourselves “Are we making burgers?”

Are we making burgers?

– T. Duckering, 2011

Anti-Pattern: The Team Who Knew Too Much

There is a phenomenon in the natural world known as “Build Monkey Affinity”, in which build engineers tend to congregate and work together, rather than integrate with the rest of society. Fascinating stuff. The trouble is, this usually leads the build monkeys to assume all the responsibilities of the CI system, because their lack of integration with the rest of the known world makes them isolated, cold and bitter (Ok, I’m going overboard here). Anyway, the point is this, if the build team don’t work with the project team, and every build task has to go through the build team, there will be a disconnect, possibly bottlenecks and a general lack of agility. Responsibility for build related activities should be devolved to the project teams as much as possible, so that bottlenecks and disconnects don’t arise. It also stops all the build knowledge from staying in one place.

Anti-Pattern: Big Ball of CI Mud

This is where you have a load of complexity in your build system, loads of obscure build scripts, multitudes of properties files, and all sorts of nonsense, all just to get a build working. It tends to happen when you over engineer your build solution because you’re compensating for a project that’s not in a fit state. I’ve worked in places where there are projects that have no regard for configuration management, project structures in source control that don’t match what they need to look like to do a build, and projects where the team have no idea what the deployed artifact should look like – so they just check all their individual work into source control and leave it for the build system to sort the whole mess out. Obviously, in these situations, you’re going to end up with some sort of crazy Heath Robinson build system which is bordering on artificial intelligence in its complexity. This is a big ball of CI mud.

Heath Robinson Build System a.k.a. "a mess"

Anti-Pattern: “They” Broke MY Build…

This is a situation that often arises when you have upstream and downstream dependencies. Let’s say your build depends on library X. Someone in another team makes a change to library X and now your build fails. This seriously blows. It happens most often when you are forced to accept the latest changes from an upstream build. This is called a “push” method. An alternative is to use the “pull” method, which is where you choose whether or not you want to accept a new release from an upstream build – i.e. you can choose to stick with the existing version that you know works with your project.

The truth is, neither system is perfect, but what would be nice is if the build system had the flexibility to be either push or pull.

The Solutions!

Fear not, for Tom has come up with some thoroughly decent solutions to some of these anti-patterns!

Project Teams Should Own Their Builds

Don’t have a separated build team – devolve the build responsibilities to the project team, share the knowledge and share the problems! Basically just buy into the whole agile idea of getting the expertise within the project team.

Project teams should involve the infrastructure team as early as possible in the project, and again, infrastructure responsibilities should be devolved to the project team as much as possible.

Have CI Experts

Have a small number of CI experts, then use them wisely! Have a program of pairing or secondment. Pair the experts with the developers, or have a rotational system of secondment where a developer or two are seconded into the build team for a couple of months. Meanwhile, the CI experts should be encouraged to go out and get a thoroughly rounded idea of current CI practices by getting involved in the wider CI community and attending meetups… like this one!

Personal Best Metrics

The trouble with targets, metrics and goals is that they can create an environment where it’s hard to take risks, for fear of missing your target. And without risks there’s less reward. Innovations come from taking the odd risk and not being afraid to try something different.

It’s also almost impossible to come up with “proper” metrics for CI. There are no standard rules, builds can’t all be under 10 minutes, projects are simply too diverse and different. So if you need to have metrics and targets, make them pertinent, or personal for each project.

Treat Your Build Environments Like They Are Production

Don’t hand crank your build environments. Sorry, I should have started with “You wouldn’t hand crank your production environments would you??” but of course, I know the answer to that isn’t always “no”. But let’s just take it as read that if you have a large production estate, to do anything other than automate the provision of new infrastructure would be very bad indeed. Tom suggests using the likes of Puppet and Chef, and here at Caplin we’re using VMWare which works pretty well for us. The point is, extend this same degree of infrastructure automation to your build and CI environments as well, make it easy to create new CI servers. And automate the configuration management while you’re at it!

Provide a Toolbox, Not a Rigid Framework

Flexibility is the name of the game here. The project teams have far more flexibility if you, as a build team, are able to offer a selection of technologies, processes and tricks, to help them create their own build system, rather than force a rigid framework on them which may not be ideal for each project. Wouldn’t it be nice, from a build team perspective, if you could allow the project teams to pick and choose whichever build language they wanted, without worrying that it’ll cause a nightmare for you? It would be great if you could support builds written in Maven, Ant, Gradle and MSBuild without any problems. This is where a toolkit comes in. If you can provide a certain level of flexibility and make your system build-language agnostic, and devolve the ownership of the scripts to the project team, then things will get done much quicker.

Consumer-Driven Contracts

It would be nice if we could somehow give upstream builds a “contract”, like a test API layer or something. Something that they must conform to, or make sure they don’t break, before they expose their build to your project. This is a sort of push/pull compromise.

And that pretty much covers it for the content of Tom’s talk. It was really well delivered, with good audience participation and the content was thought-provoking. I may have paraphrased him on the whole Jamie Oliver thing, but never mind that.

It was really interesting to hear someone so experienced in build management actually promote flexibility rather than standardisation. It’s been my experience that until recently the general mantra has been “standardise and conform!”. But the truth is that standardisation can very easily lead to inflexibility, and the cost is that projects take longer to get out of the door because we spend so much time compromising and conforming to a rigid process or framework.

Chatting to Christian Blunden a couple of months back about developer anarchy was about the first time I really thought that such a high degree of flexibility could actually be a good thing. It’s really not easy to get to a place where you can support such flexibility, it requires a LOT of collaboration with the rest of the dev team, and I really believe that secondment and pairing is a great way to achieve that. Fortunately, that’s something we do quite well at Caplin, which is pretty lucky because we’re up to 6 build languages and 4 different C.I. systems already!

What is in a name? Usually a version number, actually.

Posted on July 7, 2011 by jamesbetteley

Another fascinating topic for you – build versioning! Ok, fun it might not be, but it is important and mostly unavoidable. In an earlier blog I outlined a build versioning strategy I was proposing to use with our Java builds. Since then, the requirements have changed, as they tend to, and so I’ve had to change the versioning convention.

Essentially, what I’m after is a way of using artifact version numbers to tell me some useful at-a-glance information about the artifact I have created. Also, customers want the version number to meet their expectations – that is, when they get a new build, they want to see an easily identifiable difference in the version number between the new build and their old one. What they don’t want is a long complicated list of numbers which are hard to distinguish. For instance, it’s much easy to identify which of the following 2 versions is the latest:

5.0.1
5.0.4

but it’s not so easy to work out which of these is the latest:

5.0.1.13573
5.0.1.13753

As we’re practicing continuous delivery, any given check-in can feasibly produce a release build. So, I would like some way of identifying exactly which check in produced my builds, or at least have a way of working out which bits of source code went into my released package. There are a couple of ways we can do this:

Tag the source code – We could make the builds tag the source code in our SCM system (Perforce) with every build. This is relatively easy to do using Ant and Maven. With Ant there are numerous different ways of doing it depending on your SCM system, for instance, with subversion you need to use the SvnAnt tasks from subclipse (http://subclipse.tigris.org/svnant/svn.html) and basically perform a copy of your source url:

<copy srcUrl=”${src.url}” destUrl=”${dest.url}” message=”${version.num}”/>

(this is because tags in svn are just cheap copies with a label).

With Maven you just need to use the release plugin – this automatically handles tagging for you.

Tagging the source code is great – it keeps the version numbers as simple as I’d like, and it’s nicely traceable. However, it’s time consuming, and can result in a lot of tags. The other problem is, I can’t tell which check-in caused the build just by looking at the version number of an artifact.

Use the commit number in the build version – We use a build version of Major.Release.Patch-Build in our artifacts. The build number used to be an auto-incrementing number – this worked fine but it didn’t give us a link back to which commit had caused the build to be made. So, I decided to use the perforce changelist id (i.e. the commit version) as the build number in the version, so that builds would end up looking something like this: 1.0.0-11531.

The problem here is that the version number is not customer friendly – so I remove the build number as a final step, before the builds get released to customers. To track what version the customers have got, I still keep a record of the full build number (including the commit number) in the release notes, and I could also easily inject it into an assembly info or properties/config file if I so wished, so that customers could very easily read out the full version number just by looking in a menu somewhere.

There were several obstacles I had to overcome to get this working. The first obstacle, and really this was the one that stopped me from tagging the source code, was that the maven release plugin is abysmal when it comes to continuous delivery. I needed to use the release plugin to tag the source code, but one of the other things that the maven release plugin does is to remove the word SNAPSHOT, increment the version number, and check the pom back into source control. This would cause another build to trigger in the CI system, which in turn would increment the build number etc and cause another build to trigger – so on and so on. Basically it would create a continually building project.

So I have decided not to use the maven release plugin at all – it doesn’t seem to fit in with Continuous Delivery. In order to create potential release candidates with every successful build, I’ve removed the word SNAPSHOT from all the poms, so we aren’t making any snapshot builds anymore either (except when you build locally – more on that later). The version in the poms now takes the P4 commit number, which is injected via the Continuous Integration system, which in my case is Go. Jenkins also supports this, using the subversion plugin (if you use subversion), which sets an environment variable with the svn revision number (more details here). The Jenkins Perforce plugin does the same thing, setting the P4_CHANGELIST environment variable – so it can easily be consumed (more details here).

Go takes the P4 changelist number and puts it in an environment variable called “GO_PIPELINE_LABEL”. I read this variable in, and assign it to a property called p4.revision. I do this in the command that kicks off the build, so that it overwrites a default value which I can keep in my pom – this is useful because it means my colleagues and I don’t have to make any changes to the pom if we want to run a build locally (bear in mind if we run it locally this environment variable won’t exist on our PCs, so the build would otherwise fail). Here’s a basic run down of a sample pom, with more details to follow:

<modelVersion>4.0.0</modelVersion>
<groupId>etc.so.forth</groupId>
<artifactId>MyArtifact</artifactId>
<packaging>jar</packaging>
<version>${main.version}-${build.number}</version>

<description>Description about this application</description>

<properties>
<p4.revision>SNAP</p4.revision>
<build.number>${p4.revision}</build.number>
<main.version>5.0.2</main.version>
</properties>

<scm>
…
</scm>

<repositories>
<repository>
<snapshots>
<enabled>false</enabled>
</snapshots>
<id>release-candidate-repo</id>
<url>http://artifactory.me.com/my-rc</url>
</repository>
</repositories>

<build>
…
</build>
</project>

The value for p4.revision is “SNAP” by default, meaning that if I make a local build, I’ll get an artifact with the version 5.0.2-SNAP. I know that these builds should never be promoted to production or handed to customers because the word SNAP gives it away. However, when a build is created by the CI system, the following command is passed:

clean deploy sonar:sonar -Dp4.revision=${env.GO_PIPELINE_LABEL}

This overwrites the value for p4.revision, passing in the Perforce commit number, and the build will create something like 5.0.2-1234 (where 1234 is my imaginary p4 commit number).

I’ve added a property called main.version, which is the same as the full version but without the build number. I’ve done this so that I can package up my customer builds (ina zip) and label them with the version 5.0.2. After all, customers don’t care about the build number.

An important policy to follow is once a build is released to a customer, one of the other version numbers MUST be increased, meaning all further builds will be at least 5.0.3. The decision of which version number to increase depends on various business factors – I like to increase the 3rd number if I’m releasing a patch to a previously released build. If I’m releasing new functionality I increase the second number. The first number gets increased for major releases. The whole issue of version numbers becomes a lot less complicated if you’re in the business of releasing software to web servers and you don’t actually have to hand software over to customers. In this instance, I just keep the full version number with the build number at the end, as it’s usually someone like me who has to look after the production system anyway!

Exclude pom file from jar using maven jar plugin

Posted on June 24, 2011 by jamesbetteley

I’m using the maven jar plugin, and I want to exclude the pom file from our resultant jars. The obvious choice would be to have an “excludes” section in my jar plugin configuration, like this:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.3.1</version>
<configuration>
<excludes>
<exclude>**/pom.xml</exclude>
</excludes>
</configuration>
</plugin>

Obvious, but wrong. Then I read that there’s an “excludePomFiles” flag which you can set to true, so I tried that, and that didn’t work either.

The solution was to use the MavenArchiveConfiguration’s “AddMavenDescriptor” method, like this:

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.3.1</version>
<configuration>
<archive>
<addMavenDescriptor>false</addMavenDescriptor>
</archive>
</configuration>
</plugin>

Are Tools and Processes Stifling Our Creativity and Productivity?

Posted on June 24, 2011 by jamesbetteley

I had lunch with my friend Christian the other day (we went to wagamamas, I had Ginger Chicken Udon – delicious) and he was telling me about a company who work according to what sounded very much like developer anarchy – basically everyone is allowed to go their own route, use whatever tools or languages they please, using any framework they want to, to deliver projects. This sounded like fresh lunacy, and I couldn’t get past how much of a nightmare it must be as a build/release or sysadmin guy, to make all of that come together and then look after it.

http://centennialsociety.com/business_reply/businessreply.htm

Developer Anarchy

However, it did get me thinking about the tools and frameworks that I’ve used over the years, and whether they were too restrictive. That led me on to thinking about some of the business processes I’ve worked with, and how counter-productive they could be too. So I decided to list some of these tools, processes and frameworks, and critique them with respect to how inflexible and restrictive they can be.

Maven and Ant

Rather predictably, I’ll start with Maven. Maven is both a build tool and a framework, it uses plugins to hide the complexities of the tasks it performs “behind the scenes”. Rather than explicitly coding out what you want to compile, or what unit tests you want to run, Maven says “if you put your files here, and call this plugin, I’ll take care of it”. Most of the Maven plugins are fairly configurable, but only to an extent, and you have to consult the (very hit-and-miss) documentation every time you want to do something even slightly off the beaten path. It very much relies on the “convention over configuration” principle – as long as you conform to its convention, you’ll be fine. And it’s this characteristic which inevitably makes it so restrictive. It’s testament to how difficult it is to veer from the “Maven way” that I almost always end up using the antrun task to call an ant script to do most of the non-standard Maven things that I want to do with my builds. For instance, I currently have a requirement to copy my distributable to 2 separate repositories, one with some supporting files (which need to be filtered) and one without. This isn’t a particularly weird requirement, and you can probably do it with Maven, but despite the fact that I’ve worked with Maven on a daily basis for over 3 years, I still find it much easier to just call Ant to do stuff like this, rather than write my own plugin or go trawling through Maven’s online documentation.

Dev Team Ring-Fencing

I have worked in companies where discussion and interaction between development and QA/NetOps etc have actively been discouraged. The idea behind this insanity has been to give the developers time to do their work, without being constantly interrupted by testers, release engineers etc and so on. I can just about understand how this can happen, but I can’t really understand how someone can come to the conclusion that ring-fencing a whole team and discouraging interactions with other teams is a sustainable and sensible solution. The solution should be to find out exactly WHY the QA or NetOps team need to constantly interrupt the developers. Being told that you can’t talk freely with other teams develops business silos which can be very hard to break down, and also prevents people from getting the information they require, which can seriously impact productivity.

Audits

If I had a pound and a pat on the head every time I heard the phrase “we can’t do that, it would never get past the auditors”, by now I’d be a rich man with a flat head. Working in build and release engineering, I quite often get caught up with security and audit issues when it comes to doing deployments – what account should be used to do deployments to production? How often should passwords be changed? How do we pass the passwords? Are they encrypted? What level of access should release engineers have to the codebase? All of these questions come up at one point or another. In my experience, developers, sysadmins, managers and the like, will always come back with “you can’t do that, it’ll never get past the auditors” whenever you propose a new or slightly radical way of doing things, it’s like their safety net, and I think it’s just their way of saying “we don’t like that, it’s different, different is bad”. My suggestion is to speak with the auditors and challenge what you’ve been told. If that’s not possible, try to find out exactly what rule you’re breaking and then find out what the guidelines for compliance are on the rule in question – google is remarkably good for this, so are forums and meetups. In my experience, most of the so called compliance rules are not nearly as draconian or restrictive as many of my own colleagues would have me believe.

Meetings

Oh my god. So many meetings. I now hate pretty much all meetings. Prior to working at Caplin Systems I think my job was to attend meetings all day. I can’t really remember, I can’t actually remember a time before meetings.

The worst meetings of all are the “weekly status meetings”. I don’t know if these are widely popular but if they are, and you have to attend them, then you have my sympathies. They amount to a lot of people talking about stuff that’s already happened. Quite a lot of it has nothing to do with you, and absolutely all of it could be communicated more efficiently without needing a meeting. Also, someone takes notes. Taking notes in a meeting about stuff that’s already happened so that they can tell everyone else about what happened in a meeting about stuff that’s already happened. The pain is still raw. Don’t do this, it’s stupid. Do demos instead, they’re more interesting and we actually get to SEE stuff, rather than just hear someone droning on about stuff they’ve already done. Meetings are not productive, and they stifle my creativity by making me want to go to sleep.

MS Project

“You have to work on this today, and it has to take you eight hours, and then you have to work on a different project for 50% tomorrow, and then you have to work on bla bla bla”. No. Wrong. This is totally unrealistic. You only have to be out by 10% with your LOE (Level of Effort, or SWAG “Sophisticated Wild-Arsed Guess”) to throw the whole thing into disarray. Be sick for a day and they system is in free-fall, one project needs to shift by 1 day and suddenly you’re clashing with all these other projects and you’re assigned to work a 16 hour day. MS Project is often the tool of choice for companies who follow the Waterfall process, and it’s probably the process as much as the tool itself which causes this mess, but the tool certainly doesn’t help. I prefer to use tools that allow me to manage my time on a 2 weekly basis (like Acunote, see here), the decision of when I should work on each task is determined by me and the team I’m working with, and it has worked very successfully so far.

Source Control Systems

Aside from Visual Source Safe, I’ve never worked with a truly awful SCM. They all have really neat features and at the end of the day they perform an exceedingly important job. However, the everyday use of these tools varies from one to another, and some are more restrictive and controlling than others. For instance, Perforce, which has long been one of my favorites, only allows you to have 1 root for your client spec – this is a pain in a CI system if you can’t tell your server to swithch clientspec (or workspace). I’m currently working with Perforce coupled with Go as the Continuous Integration server, and Go forces me to check out my files to “C:\Program Files\Cruise Agent” but my build has a requirement to also sync some files to D:\. This can’t be done with Perforce because you can only have one root directory (in this case C:\).

I like SCM systems where it’s easy to create branches. I’m a big fan of personal branches, somewhere to check-in my personal work, POCs, or just incomplete bug fixes that I’d like to check-in somewhere safe before I go home. Subversion and Git make this easy, and they also allow you to merge your changes to another branch. I personally find Subversion better at this than Perforce, but that might just be me (I get confused with Perforce’s “yours” and “theirs”). SCM systems that allow for easy branching are far less restrictive, and encourage you to try something different, without having to worry about checking in to the main branch and breaking everything.

Continuous Integration Systems

C.I. systems can really get in the way of your productivity if they:

Take ages to setup a build job – How can I be productive if I’m spending so long creating a build job, or waiting for one to be made?
Keep reporting false-negatives – How can I be productive if I’m always looking into a failing build which isn’t really failing?
Don’t provide a friendly interface
Don’t provide you with the information that you need!

A good C.I. system should be a service, and a hub of information. It should be easy to copy build jobs or create new ones and it should provide all users with a single point of reference for all the build output, like test results and static analysis results. At a previous job, Pankaj Sahu, the build engineer, setup a system of automatically creating build jobs in Bamboo using Selenium and Ant, it was brilliant. Bamboo also allows you to copy jobs, which is in itself a real time saver. Jenkins also has a similar feature. Go, which is a good C.I system is slightly harder work, but it’s main advantages lay elsewhere.

Maven Assembly Plugin Filtering Part 2

Posted on June 16, 2011 by jamesbetteley

We want to copy some documents, namely a releasenote.txt, into a doc directory inside the zip file that our build creates. I’m using the assembly plugin to create the zip file, and the assembly descriptor specifies the release note file to copy and filter

The tokens I’ve got in the release notes are:

@version@
@currentdate@
@build.number@

And I have defined the values in the pom:

<currentdate>${maven.build.timestamp}</currentdate>

<build.number>${p4.revision}</build.number>

Now, if I copy the release note using the maven resource plugin, which by default copies resources to the classes output directory, the filtering works fine. This is how I defined it in the pom:

<resources>
<resource>
<directory>src/main/resources</directory>
<filtering>true</filtering>
<includes>
<include>RELEASENOTE.txt</include>
</includes>
</resource>
</resources>

Unfortunately I don’t want to copy it to the classes directory, I want it to go in the zip assembly, in a docs directory, hence the definition in the assembly descriptor (called kit.xml). The full kit.xml can be seen at the end of this post. So, the suggested thing to do is specify the file you want to copy, and set <filtered> to true, like this:

<file>
<source>src/main/resources/RELEASENOTE.txt</source>
<outputDirectory>doc</outputDirectory>
<filtered>true</filtered>
</file>

Of course, it doesn’t work. And here’s why – unlike the maven resource plugin, the assembly plugin’s filtering doesn’t seem to work with tokens in the @bla@ format, but instead insists on using ${bla} instead. And not only that, but it won’t read my properties and values that I’ve defined in my pom either, it requires you to create a filter.properties file which defines variables/values in the style:

variable1=value1

variable2=-value2

Obviously I don’t want to do that, I’ve already got them defined in my pom and I hate duplication (think DRY principle!). So that left me with 2 options. I could explicitly define the maven resource plugin, like this:

<plugin>
        <artifactId>maven-resources-plugin</artifactId>
        <version>2.5</version>
        <executions>
          <execution>
            <id>copy-resources</id>
            <phase>prepare-package</phase>
            <goals>
              <goal>copy-resources</goal>
            </goals>
            <configuration>
              <outputDirectory>some/other/dir</outputDirectory>
              <resources>
                <resource>
                  <directory>src/main/resources</directory>
                  <filtering>true</filtering>
                </resource>
              </resources>
            </configuration>
          </execution>
        </executions>
      </plugin>

and tell it to copy my releasenote.txt file to some directory, and do the filtering on it in the process, and then bind it to a phase which happens before my assembly plugin is called (as shown above). I would then need to change my assembly descriptor to tell it where to find the new, properly filtered releasenote.txt file and copy it to the doc dir, like this:

<file>
<source>src/main/resources/RELEASENOTE.txt</source>
<outputDirectory>doc</outputDirectory>
<filtered>true</filtered>
</file>

An alternative, and this is the one I went for, is to let the maven resource filtering copy the file from src/main/resources into the target/classes directory as it does by default, and then tell the assembly descriptor to copy it from there to the doc directory:

<file>
<source>src/main/resources/RELEASENOTE.txt</source>
<outputDirectory>doc</outputDirectory>
</file>

It’s far from ideal and it’s very annoying that the maven assembly plugin does filtering differently, but in the end it gets the job done.

My full assembly descriptor, kit.xml:

<assembly>
<id>kit</id>
<formats>
<format>zip</format>
</formats>
<fileSets>
<fileSet>
<directory>src/main/resources</directory>
<outputDirectory>doc</outputDirectory>
<includes>
<include>*.*</include>
</includes>
<excludes>
<exclude>latest.properties</exclude>
<exclude>RELEASENOTE.txt</exclude>
</excludes>
</fileSet>
</fileSets>
<files>
<file>
<source>target/classes/RELEASENOTE.txt</source>
<outputDirectory>doc</outputDirectory>
<filtered>true</filtered>
</file>
<file>
<source>target/${artifactId}-${version}.${packaging}</source>
<outputDirectory>/</outputDirectory>
<destName>container-filtering-module.jar</destName>
</file>
</files>
</assembly>

Maven Release Plugin and Continuous Delivery

Posted on June 15, 2011 by jamesbetteley

I was setting up a Continuous Delivery system using Maven as the build tool, Perforce as the SCM and Go (ThoughtWorks’ CI system). All was going perfectly well until I got to the point when I no longer wanted to make snapshot builds…

The idea behind my Continuous Delivery system was this:

Every check-in runs a load of unit tests
If they pass it runs a load of acceptance tests
If they pass we run more tests – Integration, scenario and performance tests
If they all pass we run a bunch of static analysis and produce pretty reports and eventually deploy the candidate to a “Release Candidate” repository where QA and other like-minded people can look at it, prod it, and eventually give it a seal of approval.

As you can see, there’s no room for the notion of “snapshot” and “release” builds being separate here. Every build is a potential release build. So, a few days ago I went right ahead and used the maven release plugin, and that was the last time I remember smiling, having fun, getting a full night’s sleep, and my brain not hurting.

The problem is this: the maven release plugin doesn’t really work for continuous delivery. And what’s more, it REALLY doesn’t work with Go and Perforce. I’ll start with the Go/Perforce issues: I got loads of errors thanks to the way Go runs as the system user, and creates its own clientspecs. The results of this debacle are detailed here and here.

I managed to finally get around the clientspec/P4/Go problems with some help from my colleague Toby Catlin who bears the scars of similar skirmishes with Go and Perforce from days gone by. The “fix” was to create a perforce config file and an “uber” client spec. The perforce config file specified the uber clientspec and it lived in the root of the project directory. It was hardly a satisfactory workaround, as it meant that every project would need to have this file, and the uber clientspec would need to be updated every time a new build job was created. But never mind that, it was just a relief to see the builds going green for a change.

And that’s when it happened… the release build completed. The maven release plugin increased the version number in the pom and checked it in. And then Go detected the change in the pom and checked it out again and started building again. This then updated the version number and checked it in, which in turn got detected and kicked off another build CAN YOU SEE THE GLARINGLY OBVIOUS PROBLEM HERE????

It’s obvious really. I’ve always made my maven release builds a manual process in the past and that was exactly why, I’d just forgotten all about it. So, I’ve decided not to use the maven release plugin at all. Every build now just creates a “release” build because I’ve removed all instances of the word SNAPSHOT from the poms. If they pass all their tests and look good enough, they’re automatically promoted to the release candidate repository. And everyone’s happy. Also, I’ve added a property to the builds which pulls in a variable from the Go system, and if that’s not present the “deploy to release candidate repository” step fails – this is to stop developers from manually creating releases – all release builds must come from the CI system.

Maven Release Plugin and Perforce Clientspecs

Posted on June 14, 2011 by jamesbetteley

I’m getting a very annoying error trying to do maven release builds on our CI servers, which isn’t appearing on my local workstation. The build seems to fail because the pom file is under source control (I’m using Perforce) and so it’s read-only. However, It should check out the pom file so that it isn’t read only. Alas, that doesn’t seem top be working. Here are the errors I got initially:

[ERROR] BUILD ERROR
[INFO] ————————————————————————
[INFO] Error writing POM: C:\Program Files\Cruise Agent\pipelines\yadda\yadda\pom.xml (Access is denied)

The reason behind it appeared to be:

[ERROR] CommandLineException Exit code: 1 – Client ‘xpcruisebuildvm1-STANDARD-ALL’ can only be used from host ‘xpcruisebuildvm1’.

Command line was:p4 -d “C:\Program Files\Cruise Agent\pipelines\yadda\yadda” -p perforce.mycompany.com:1666 ed
it pom.xml
org.codehaus.plexus.util.cli.CommandLineException: Exit code: 1 – Client ‘xpcruisebuildvm1-STANDARD-ALL’ can only be used from host ‘xpcruisebuildvm1’

So the first thing I did was create a new clientspec for that machine which had all the files mapped to its c:\temp directory, and then tried to run the build again, assuming this would fix the issue but present me with a whole new problem about how to fit this all in to my CI system. However, this also failed:

[ERROR] BUILD ERROR
[INFO] ————————————————————————
[INFO] Error writing POM: C:\TEMP\yadda\pom.xml (Access is denied)

This is confusing because the clientspec I’m using does have these files in its view and therefore Maven should be able to edit them. Also, this is how I have it setup on my local workstation, and that works fine…

After a lot of trial and error I managed to make some progress. One of the issues was that the client spec had the following mapping in it:

//JavaDevelopment/main/yadda/… //XPCRUISEVM808_release/temp/yadda/…

and the root was set to C:\

This means all P4 files should sync to my C:\TEMP directory, which already existed on the machine. As expected, the sync worked fine and all files appeared in C:\TEMP\yadda.

And therein lies the rub: the client spec uses lowercase “temp”, while the windows directory was uppercase “TEMP”. As simple as that. I changed the client spec to math the filesystem and for good measure updated my release plugin version to 2.1. Problem solved (well, actually that presents me with a whole new problem because I don’t want every build agent to have different client specs, because I’m using Go, and that puts them in it’s own directory under the name of the build job. Grrrr).

POM ‘ release:prepare release’ not found in repository

Posted on June 13, 2011 by jamesbetteley

I’m rather irritatingly getting this error in my maven builds at the moment, trying to setup some release builds using Go:

POM ' release:prepare release' not found in repository

One issue I have with Go is that it doesn’t natively support maven. This isn’t really much of a big deal because I can just tell it to run a custom command, and point it to the mvn shell or batch file (this could be a bit of a pain if I want my builds run on windows and/or linux but don’t want to have to define separate build jobs for each one, but I don’t, and I can’t think of any reason why I would, so that’s ok). Anyway, the issue this time is with the way I setup the build job. I used the new (in version 2.2) clicky-UI to setup the job, like telling it to run the mvn batch file, and what arguments to pass. This just seemed not to work. When I looked at the Go xml file it looked a bit like this:

<job name=”build_release”>

<tasks>

<exec command=”D:\buildTools\maven\2.2.1\bin\mvn.bat” workingdir=”yadda\yadda”>

<args>-B release:prepare release:perform</args>

</tasks>

<resources>

<resource>windows</resource>

</resources>

</job>

So I deleted it and manually edited the xml, making it look like this instead:

<job name=”build_release”>

<tasks>

<exec command=”D:\buildTools\maven\2.2.1\bin\mvn.bat” args=”-B release:prepare release:perform” workingdir=”yadda\yadda” />

</tasks>

<resources>

<resource>windows</resource>

</resources>

</job>

And this seems to have fixed it. Not very impressive at all.