Dependency report causing maven builds to run slowly

Issue:

Maven builds running unusually slowly when using site-deploy. Dependency report taking a long time to generate. Url is invalid – repository will be blacklisted.

Cause:

By default, dependency location checking is enabled, and maven will try to connect to the dependency repository URLs for each external dependency. If it cannot connect to those URLs for whatever reason, it’ll time out – but not until after a long pause, sometimes up to 10 minutes for each URL.

Fix:

One way of speeding up these builds is to bypass the whole dependency checking process. You can do this by defining the maven-project-info-reports plugin in your pom, and setting dependencyLocatiosEnabled to false:

 

<plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-project-info-reports-plugin</artifactId>

<configuration>

<dependencyLocationsEnabled>false</dependencyLocationsEnabled>

</configuration>

</plugin>

 

The only drawback of this fix is that the dependency repository locations won’t be published in your maven site report.

Maven sites for failed builds

I’m using Bamboo and Maven to build our code, and Cobertura as the unit test coverage checking tool. I’ve setup the builds to fail if there are less than, say, 70% unit test coverage. However, I still want the maven site to be created so that I can go around waving it in people’s faces. The trouble is, if there’s less than 70% coverage, the build fails and therefore no cobertura report is created in the maven site! Gah!

The good thing about the maven sites is that they’re easy for whole project teams to understand, not just devs and other technical people. If a build failed because there’s less than 70% unit test coverage I currently have to look in the bamboo build log to see exactly why it failed (I know I also get the emails but there are way too many of them for it to be manageable).

In short – is there a way of producing the cobertura code coverage reports even if the build fails?

Fun with FindBugs

We’ve just moved to a new master pom file in an effort to make our lives a bit easier, and to allow certain builds to carry on using the old master pom file which was basically quite rubbish. You see, the old master pom file just defined a load of plugins, mostly in the plugin management section, so they had to be referenced by the application poms anyway. The idea with the new master pom is that it enforces the use of certain plugins, and ALSO enfiorces certain standards for builds to pass – for instance, we included a cobertura coverage rate of 80% and made the builds fail if there were any findbugs issues. It sounded like a good idea at the time. So, we put these plugin definitions directly into the build section of the pom, like so:

<build>

<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>

<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.4</version>
<configuration>
<testClassesDirectory>
build/maven/${artifactId}/target/test-classes
</testClassesDirectory>
</configuration>
</plugin>

<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<version>2.4</version>
<configuration>
<formats>
<format>html</format>
<format>xml</format>
</formats>
<check>
<totalBranchRate>54</totalBranchRate>
<totalLineRate>75</totalLineRate>
</check>
</configuration>
<executions>
<execution>
<goals>
<goal>clean</goal>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-pmd-plugin</artifactId>
<version>2.5</version>
<executions>
<execution>
<goals>
<goal>check</goal>
<goal>cpd-check</goal>
</goals>
</execution>
</executions>
<configuration>
<linkXref>true</linkXref>
<targetJdk>1.6</targetJdk>
<sourceEncoding>utf-8</sourceEncoding>
<failOnViolation>false</failOnViolation>
<outputDirectory>build/maven/${pom.artifactId}/target/pmd-reports</outputDirectory>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>findbugs-maven-plugin</artifactId>
<version>2.3.1</version>
<!– NOT USING THIS YET
<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions> –>
</plugin>
</plugins>
</build>

The first thing I had to do (as you can see) is disable the check goal because it was causing findbugs to fail pretty much every build. The next thing I had to do was remove the whole of the following configuration section from the findbugs plugin:

<configuration>
<threshold>High</threshold>
<effort>Max</effort>
<xmlOutput>true</xmlOutput>
<xmlOutputDirectory>build/maven/${artifactId}/target/site</xmlOutputDirectory>
</configuration>

I’ve kept this section in the reporting section though.

The next thing I got was this error:

03-Nov-2010 09:53:03 [java] Java Result: 1
03-Nov-2010 09:53:03 [Fatal Error] findbugsTemp.xml:1:1: Premature end of file.
03-Nov-2010 09:53:03 [INFO] ————————————————————————
03-Nov-2010 09:53:03 [ERROR] FATAL ERROR
03-Nov-2010 09:53:03 [INFO] ————————————————————————
03-Nov-2010 09:53:03 [INFO] Premature end of file.
03-Nov-2010 09:53:03 [INFO] ————————————————————————
03-Nov-2010 09:53:03 [INFO] Trace
03-Nov-2010 09:53:03 org.xml.sax.SAXParseException: Premature end of file.

and that was fixed by changing this bit of the findbugs plugin configuration:

<effort>Max</effort>

to this:

<effort>Default</effort>

So, in short, using “Max” gives us java out of memory exceptions. Which is not exactly convenient. Not sure how to fix this though.

Best Practices for Build and Release Management Part 2

Ok, as promised in Part 1, I’ll go into a bit more detail about each of the areas outlined previously, starting with…

The Build Process

This area, perhaps more than any other area I’ll be covering in this section, has benefited most from the introduction of some ultra handy tools. Back in the day, building/compiling software was fairly manual, and could only be automated to a certain degree, make files and batch systems were about as good as it got, and even that relied on a LOT of planning and could quite often be a nightmare to manage.

These days though, the build phase is exceedingly well catered for and is now a very simple process, and what’s more, we can now get an awful lot more value out of this single area.

As I mentioned before, one of the aims of release management is to make software builds simple, quick and reliable. Tools such as Ant, Nant (.Net version of Ant), Maven, Rake and MSBuild help us on our path towards our goal in many ways. Ant, MSBuild and Nant are very simple XML based scripting languages which offer a wide ranging level of control – for instance, you can build entire solutions with a single line of script, or you can individually compile each project and specify each dependency – it’s up to you to decide what level of control you need. I believe that build scripts should be kept simple and easy to manage, so when dealing with NAnt and MSBuild for .Net solutions I like to build each project by calling an .proj file rather than specifically compiling each library. The .proj files should be constructed correctly and stored in source control. Each build should get the latest proj file  (and the rest of the code, including shared libraries – more on that later) and compile the project.

For Java projects. Ant and Maven are the most popular tools. Ant, like Nant, gives the user a great deal of control, while Maven has less inherent flexibility and enforces users to adhere to its processes. However, both are equally good at helping us make our build simple, quick and reliable. Maven uses POM files to control how projects are built. Within these POM files a build engineer will define all the goals needed to compile the project. This might sound a little tedious but the situation is made easier by the fact that POM files can inherit from master/parent POM files, reducing the amount of repetition and keeping your project build files smaller, cleaner and easier to manage. I would always recommend storing as much as possible in parent POM files, and as little as you can get away with in the project POMs.

One of the great improvements in software building in recent years has been the introduction of Continuous Integration. The most popular CI tools around are CruiseControl, CruiseControl.Net, Hudson and Bamboo. In their simplest forms, CI tools are basically just schedulers, and they essentially just kick off your build tools. However, that’s just the tip of the iceberg, because these tools can do much, MUCH more than that – I’ll explain more later, but for now I’ll just say that they allow us to do our builds automatically, without the need for any human intervention. CI tools make it very easy for us to setup listeners to poll our source code repositories for any changes, and then automatically kick off a build, and then send us an email to let us know how the build went. It’s very simple stuff indeed.

So let’s take a look at what we’ve done with our build process so far:

  • We’ve moved away from manually building projects and started using simple build scripts, making the build process less onerous and not so open to human error. Reliability is on the up!
  • We’ve made our build scripts as simple as possible – no more 1000 line batch files for us! Our troubleshooting time has been significantly reduced.
  • We’ve moved away from using development UIs to make our builds – our builds are now more streamlined and faster.
  • We’ve introduced a Continuous Integration system to trigger our builds whenever a piece of code is committed – our builds are now automated.

So in summary, we’ve implemented some really simple steps and already our first goal is achieved – we’ve now got simple, quick and reliable builds. Time for a cup of tea!