Archive

Posts Tagged ‘Maven’

Maven Release Plugin and Continuous Delivery

June 15, 2011 7 comments

I was setting up a Continuous Delivery system using Maven as the build tool, Perforce as the SCM and Go (ThoughtWorks’ CI system). All was going perfectly well until I got to the point when I no longer wanted to make snapshot builds…

The idea behind my Continuous Delivery system was this:

  • Every check-in runs a load of unit tests
  • If they pass it runs a load of acceptance tests
  • If they pass we run more tests – Integration, scenario and performance tests
  • If they all pass we run a bunch of static analysis and produce pretty reports and eventually deploy the candidate to a “Release Candidate” repository where QA and other like-minded people can look at it, prod it, and eventually give it a seal of approval.

As you can see, there’s no room for the notion of “snapshot” and “release” builds being separate here. Every build is a potential release build. So, a few days ago I went right ahead and used the maven release plugin, and that was the last time I remember smiling, having fun, getting a full night’s sleep, and my brain not hurting.

The problem is this: the maven release plugin doesn’t really work for continuous delivery. And what’s more, it REALLY doesn’t work with Go and Perforce. I’ll start with the Go/Perforce issues: I got loads of errors thanks to the way Go runs as the system user, and creates its own clientspecs. The results of this debacle are detailed here and here.

I managed to finally get around the clientspec/P4/Go problems with some help from my colleague Toby Catlin who bears the scars of similar skirmishes with Go and Perforce from days gone by. The “fix” was to create a perforce config file and an “uber” client spec. The perforce config file specified the uber clientspec and it lived in the root of the project directory. It was hardly a satisfactory workaround, as it meant that every project would need to have this file, and the uber clientspec would need to be updated every time a new build job was created. But never mind that, it was just a relief to see the builds going green for a change.

And that’s when it happened… the release build completed. The maven release plugin increased the version number in the pom and checked it in. And then Go detected the change in the pom and checked it out again and started building again. This then updated the version number and checked it in, which in turn got detected and kicked off another build CAN YOU SEE THE GLARINGLY OBVIOUS PROBLEM HERE????

It’s obvious really. I’ve always made my maven release builds a manual process in the past and that was exactly why, I’d just forgotten all about it. So, I’ve decided not to use the maven release plugin at all. Every build now just creates a “release” build because I’ve removed all instances of the word SNAPSHOT from the poms. If they pass all their tests and look good enough, they’re automatically promoted to the release candidate repository. And everyone’s happy. Also, I’ve added a property to the builds which pulls in a variable from the Go system, and if that’s not present the “deploy to release candidate repository” step fails – this is to stop developers from manually creating releases – all release builds must come from the CI system.

Maven Release Plugin and Perforce Clientspecs

June 14, 2011 1 comment

I’m getting a very annoying error trying to do maven release builds on our CI servers, which isn’t appearing on my local workstation. The build seems to fail because the pom file is under source control (I’m using Perforce) and so it’s read-only. However, It should check out the pom file so that it isn’t read only. Alas, that doesn’t seem top be working. Here are the errors I got initially:

[ERROR] BUILD ERROR
[INFO] ————————————————————————
[INFO] Error writing POM: C:\Program Files\Cruise Agent\pipelines\yadda\yadda\pom.xml (Access is denied)

The reason behind it appeared to be:

[ERROR] CommandLineException Exit code: 1 – Client ‘xpcruisebuildvm1-STANDARD-ALL’ can only be used from host ‘xpcruisebuildvm1′.

Command line was:p4 -d “C:\Program Files\Cruise Agent\pipelines\yadda\yadda” -p perforce.mycompany.com:1666 ed
it pom.xml
org.codehaus.plexus.util.cli.CommandLineException: Exit code: 1 – Client ‘xpcruisebuildvm1-STANDARD-ALL’ can only be used from host ‘xpcruisebuildvm1′

So the first thing I did was create a new clientspec for that machine which had all the files mapped to its c:\temp directory, and then tried to run the build again, assuming this would fix the issue but present me with a whole new problem about how to fit this all in to my CI system. However, this also failed:

[ERROR] BUILD ERROR
[INFO] ————————————————————————
[INFO] Error writing POM: C:\TEMP\yadda\pom.xml (Access is denied)

This is confusing because the clientspec I’m using does have these files in its view and therefore Maven should be able to edit them. Also, this is how I have it setup on my local workstation, and that works fine…

After a lot of trial and error I managed to make some progress. One of the issues was that the client spec had the following mapping in it:

//JavaDevelopment/main/yadda/… //XPCRUISEVM808_release/temp/yadda/…

and the root was set to C:\

This means all P4 files should sync to my C:\TEMP directory, which already existed on the machine. As expected, the sync worked fine and all files appeared in C:\TEMP\yadda.

And therein lies the rub: the client spec uses lowercase “temp”, while the windows directory was uppercase “TEMP”. As simple as that. I changed the client spec to math the filesystem and for good measure updated my release plugin version to 2.1. Problem solved (well, actually that presents me with a whole new problem because I don’t want every build agent to have different client specs, because I’m using Go, and that puts them in it’s own directory under the name of the build job. Grrrr).

POM ‘ release:prepare release’ not found in repository

June 13, 2011 3 comments

I’m rather irritatingly getting this error in my maven builds at the moment, trying to setup some release builds using Go:

POM ' release:prepare release' not found in repository

One issue I have with Go is that it doesn’t natively support maven. This isn’t really much of a big deal because I can just tell it to run a custom command, and point it to the mvn shell or batch file (this could be a bit of a pain if I want my builds run on windows and/or linux but don’t want to have to define separate build jobs for each one, but I don’t, and I can’t think of any reason why I would, so that’s ok). Anyway, the issue this time is with the way I setup the build job. I used the new (in version 2.2) clicky-UI to setup the job, like telling it to run the mvn batch file, and what arguments to pass. This just seemed not to work. When I looked at the Go xml file it looked a bit like this:

<job name=”build_release”>

<tasks>

<exec command=”D:\buildTools\maven\2.2.1\bin\mvn.bat” workingdir=”yadda\yadda”>

<args>-B release:prepare release:perform</args>

</tasks>

<resources>

<resource>windows</resource>

</resources>

</job>

So I deleted it and manually edited the xml, making it look like this instead:

<job name=”build_release”>

<tasks>

<exec command=”D:\buildTools\maven\2.2.1\bin\mvn.bat” args=”-B release:prepare release:perform” workingdir=”yadda\yadda” />

</tasks>

<resources>

<resource>windows</resource>

</resources>

</job>

And this seems to have fixed it. Not very impressive at all.

Build Versioning Strategy

June 9, 2011 3 comments

Over the last few years I’ve followed a build versioning strategy of the following format:

<Major Version>.<Release Version>.<Patch Number>.<Build ID>

The use of decimal points allows us to implement an auto-incrementing strategy for our builds, meaning the Build ID doesn’t need to be manually changed each time we produce a build, as this is taken care of by the build system. Both Maven and Ant have simple methods of incrementing this number.

Ensuring that each build has a unique version number (by incrementing the Build ID) allows us to distinguish between builds, as no two builds of the same project will have the same BuildID. The other numbers are changed manually, as and when required.

When to Change Versions:

Major Version – Typically changes when there are very large changes to product or project, such as after a rewrite, or a significant change to functionality

Release Version – Incremented when there is an official release of a project which is not considered a Major Version change. For example, we may plan to release a project to a customer in 2 or 3 separate releases. These releases may well represent the same major version (say version 5) but we would still like to be able to identify the fact that these are subsequent planned releases, and not patches.

Patch Number – This denotes a patch to an existing release. The release that is being patched is reflected in the Release Version. A patch is usually issued to fix a critical bug or a collection of major issues, and as such is different to a “planned” release.

Build ID – This auto-increments with each release build in the CI system. This ensures that each build has a unique version number. When the Major Version, Release Version or Patch Number is increased, the Build ID is reset to 1.

Examples:

17.23.0.9 – This represents release 17.23. It is the 9th build of this release.

17.24.0.1 – This is the next release, release 17.24. This is the first build of 17.24.

17.24.1.2 – This represents a patch for release 17.24. This is the first patch release, and happens to be the 2nd build of that patch.

Maven Assembly Plugin Filtering

A colleague at Caplin made some changes to a build pom to setup some filtering on some files. We followed the instructions given on the maven site here. It basically tells you to list the files you want to filter in your assembly descriptor, and to then list the filter and then configure your filter file inside your assembly plugin, like this:

      <plugin>

<artifactId>maven-assembly-plugin</artifactId>

<version>2.2.1</version>

<configuration>

<filters>

<filter>src/assemble/filter.properties</filter>

</filters>

<descriptors>

<descriptor>src/assemble/distribution.xml</descriptor>

</descriptors>

</configuration>

</plugin>

If you do this though, you’ll get an error saying something like:

[INFO] Error configuring: org.apache.maven.plugins:maven-assembly-plugin.

Reason: ERROR: Cannot override read-only parameter: filters in goal: assembly:single

This is basically because the Maven documentation is wrong. In reality you need to add the filters section as a child of the build element, not a child of the assembly plugin’s configuration element.

So, it should look more like this:

    <build>
<filters>
<filter>src/assemble/filter.properties</filter>
</filters>
<plugins>

<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2-beta-1</version>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<descriptors>
<descriptor>src/main/assembly/kit.xml</descriptor>
</descriptors>
<finalName>${pom.artifactId}-${pom.version}-${p4.revision}</finalName>
<outputDirectory>build/maven/${pom.artifactId}/target</outputDirectory>
<workDirectory>build/maven/${pom.artifactId}/target/assembly/work</workDirectory>
</configuration>
</execution>
</executions>
</plugin>

</plugins>

</build>

Changing a filename using the maven assembly plugin

I’m currently working on a project which requires the build to produce a zip archive of the jar and some other stuff (doc files mainly). I’ve used the maven assembly plugin, with an assembly descriptor to help me out here. However, there was one slightly unusual requirement – I needed to change the name of the jar so that inside the zip, it has a non-standard name, and no version number (the reasons behind this are fairly weak, but basically it’s because a load of other scripts, which we can’t change, expect to find the jar in this non-standard format).

So, the build produces:

myproject-1.0.0.0-SNAPSHOT.jar

but in the zip I need to have:

my-project.jar

Here’s how I did it.

  • Include the maven-assembly plugin in the build:

<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2-beta-1</version>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<descriptors>
<descriptor>src/main/assembly/kit.xml</descriptor>
</descriptors>
<finalName>${pom.artifactId}-${pom.version}</finalName>
<outputDirectory>build/maven/${pom.artifactId}/target</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>

  • Create the kit.xml in src/main/assembly and specify <destName> in the file inclusion. Here’s my kit.xml:

<assembly>
<id>kit</id>
<formats>
<format>zip</format>
</formats>
<fileSets>
<fileSet>
<directory>src/main/resources</directory>
<outputDirectory>doc</outputDirectory>
<includes>
<include>*.*</include>
</includes>
</fileSet>
</fileSets>
<files>
<file>
<source>build/maven/${artifactId}/target/${artifactId}-${version}.${packaging}</source>
<outputDirectory>/</outputDirectory>
<destName>my-project.jar</destName>
</file>
</files>
</assembly>

As you can see, the assembly descriptor has a “files” section, which is what does the trick for us. I’ve isolated the actual section below for clarity:

<files>
<file>
<source>build/maven/${artifactId}/target/${artifactId}-${version}.${packaging}</source>
<outputDirectory>/</outputDirectory>
<destName>my-project.jar</destName>
</file>
</files>

Maven assembly plugin inheritance headache

Today I’ve had a headache with the maven assembly plugin, and the way it inherits from a parent. The story goes as follows:

I have a uber parent pom, which defines the assembly plugin in the pluginManagement section, and it looks like this:

<artifactId>maven-assembly-plugin</artifactId>
<version>2.2-beta-1</version>
<configuration>
<descriptors>
<descriptor>src/assembly/kit.xml</descriptor>
</descriptors>
<finalName>${pom.artifactId}-${pom.version}</finalName>
<outputDirectory>build/maven/${pom.artifactId}/target</outputDirectory>
<workDirectory>build/maven/${pom.artifactId}/target/assembly/work</workDirectory>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>

Then I have a parent pom for a collection of projects, and this parent pom has the following definition in it:

<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>attached</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptors>
<descriptor>src/main/assembly/kit.xml</descriptor>
</descriptors>
</configuration>

Now, if I simply put <artifactId>maven-assembly-plugin</artifactId> in one of the module’s pom files, it inherits most of everything from the project parent, which makes sense to me. The problem arises when we want to do something nifty with the assembly plugin, namely, create a jar with dependencies, and THEN include that in the package as described by an assembly descriptor which is different to the parent.

Here’s what I had in my project pom:

<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>attached</goal>
</goals>
</execution>
<execution>
<id>copy-fields-conf-file</id>
<phase>package</phase>
<goals>
<goal>attached</goal>
</goals>
</execution>
</executions>
<configuration>
<descriptors>
<descriptor>src/main/assembly/kit.xml</descriptor>
</descriptors>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifestEntries>
<build-number>${build-number}</build-number>
</manifestEntries>
</archive>
</configuration>

The problem was that the initial make-assembly execution was inheriting its configuration from the parent, which tells it there’s an assembly descriptor in src/main/assembly/kit.xml. That file DOES exist there, but inside it, it has:

<file>
<source>build/maven/permissioning-auth-module/target/permissioning-auth-module-${project.version}-jar-with-dependencies.jar</source>
<outputDirectory>/</outputDirectory>
</file>

Now, this is a problem because this jar hasn’t been created yet, that’s what we’re trying to create! So, I commented this section out from the parent, only to find that it inherits it from the uber parent, and here’s why:

In the assembly plugin definition you have execution sections and configurations. You can tie a configuration in to an execution phase, by simply adding it inside the execution scope. If you don’t, and you leave it outside, then it becomes a general definition which is inherited by any child projects whenever they call the assembly plugin and don’t override it explicitly. This is usually fine, but not when you don’t want to declare an assembly descriptor. Because the asembly descriptor is defined in the parent(s), it always goes looking for it even if you don’t want it to in your execution (which we don’t). There’s a workaround: create a blank assembly descriptor and point your execution at that, but that’s not very elegant. The trick is to always tie your configurations in to an execution phase, so the parent pom(s) end up looking something more like this:

<artifactId>maven-assembly-plugin</artifactId>
<version>2.2-beta-1</version>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<descriptors>
<descriptor>src/assembly/kit.xml</descriptor>
</descriptors>
<finalName>${pom.artifactId}-${pom.version}</finalName>
<outputDirectory>build/maven/${pom.artifactId}/target</outputDirectory>
<workDirectory>build/maven/${pom.artifactId}/target/assembly/work</workDirectory>
</configuration>
</execution>
</executions>

And the module’s pom can now look like this:

<artifactId>maven-assembly-plugin</artifactId>
<executions>
<execution>
<id>pack-assembly</id>
<phase>prepare-package</phase>
<goals>
<goal>single</goal>
</goals>
<configuration>
<descriptors>
</descriptors>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifestEntries>
<build-number>${build-number}</build-number>
</manifestEntries>
</archive>
</configuration>
</execution>
<execution>
<id>copy-fields-conf-file</id>
<phase>package</phase>
<goals>
<goal>attached</goal>
</goals>
<configuration>
<descriptors>
<descriptor>src/main/assembly/kit.xml</descriptor>
</descriptors>
</configuration>
</execution>
</executions>

Headache over :-)

Follow

Get every new post delivered to your Inbox.

Join 424 other followers