A great way to corrupt a .war file

So, yesterday I had an interesting issue with a corrupt .war file which was producing all sorts of errors when I tried to extract it, for instance:

BUILD FAILED
java.lang.RuntimeException: data starting at 0 is in unknown format

When I manually extracted the war file, or did an inspection on it (using “unzip -T”), then it gave these errors:

error:  invalid compressed data to inflate

It turns out I was corrupting it by running a filter on it at an earlier stage in the build, a bit like this:

<copy todir=”${release.dir}/${app.name}/tomcat/” includeEmptyDirs=”false”>
<fileset dir=”${build.dir}/${app.name}/tomcat/”>
</fileset>
<filterset>
<filter token=”SERVERNAME” value=”${dest}” />
<filter token=”DBSERVER” value=”${db.server}” />
</filterset>
</copy>

So basically I had to make sure that the war file was copied separately, and then the other files (which i wanted to run the filter on) were copied afterwards. Interesting…

Best Practices for Build and Release Management Part 3: Making software deployments rapid, reliable and repeatable

When we do our deployments, it’s usually at that time in the project everyone in the world is eager for something to be delivered. With all eyes on you, the last thing you want is a complicated, arduous, risk-filled deployment task. What you need is a rapid, reliable and repeatable task that you have the utmost faith in.

Of course, the simple solution in a nutshell is to automate deployments. This easier said than done of course, but it’s also not rocket science.

If I think of some of the deployments I’ve done in the past and break them down into their constituent manual steps, I might come up with something along these lines:

  1. Download a tar ball from a repository
  2. Extract the tar and move some files.
  3. Make changes to the config files
  4. Add some third party files (maybe tomcat for instance)
  5. Tar it back up
  6. Prep target servers by removing the existing version
  7. Send the new version to each server
  8. Extract it on each host
  9. Start the application
  10. Test that it works

That’s quite a few steps, and in reality each one of these steps would probably consist of several others.

These tasks might not seem risk-filled and arduous but in actual fact they are. If all these steps were manual, then there’d be ample opportunity for human error. Perhaps the downloading of the tar file was incomplete – we’ve introduced a machine error that might not be caught until step 10!

What about the configurations? We might have different configurations for each locale. In a manual process we might have to do this by hand. The opportunity for human error is greatly extended, and mistakes in config files can be costly.

And what about the sheer amount of time it would take if we tried to do these tasks ourselves? I know I’d rather be getting on with something else more constructive!

So let’s see what we can do about making this deployment rapid, reliable and repeatable.

In actual fact we can easily automate all these steps.

We can write a script to download the tar file and then check it against an md5 checksum. An Ant script could do this for us nicely, and hey presto, we’ve removed the risk of that machine error.

Here’s an Ant code snippet for getting a tar file from a maven repository (handy if you use Maven ;-)) and doing an md5 check:

<target name=”get_tar”>
<echo>Getting the distribution from ${sourcehost}</echo>
<scp file=”${sourcefile}” todir=”${tars.dir}/${app.name}/${filename}” trust=”true” keyfile=”${key}” passphrase=” “/>
</target>

Next Step – extracting the file and moving some files about: Really we should look to minimise the amount of file moving we need to do, if the application doesn’t get delivered in the right structure then this should be addressed earlier in the process so that it does get delivered in exactly the right structure for our production system. Go back to the developer/vendor and tell them how you would expect the delivery to look. If this isn’t an option, I’ve again used ant tasks to get my releases into the state I want them in. Here’s an example:

<!– copy the tomcat files into the release –>

<copy todir=”${release.dir}/supportapps/java/${jre_jdk.version}” includeEmptyDirs=”false”>

<fileset dir=”${supportapps.dir}/${jre_jdk.version}”>

</fileset>

</copy>

<!– copy the application configs files into the right directory –>

<copy todir=”${release.dir}/config” overwrite=”true” includeEmptyDirs=”false”>

<fileset dir=”${config.template.dir}/” />

<filterset>

<filter token=”VERSION” value=”${version.num}” />

<filter token=”SERVERNAME” value=”${destination.name}” />

<filter token=”DBNAME” value=”${dbname}” />

<filter token=”UID” value=”${username}” />

</filterset>

</copy>

As you can see, I’ve also made some changes to the configs files here while I copied them from one directory to another. In theory, the only difference between an application deployed on a test environment, and the same application deployed on the production environment, should be the config files. In reality we also have databases which can affect the functionality of our system, but we shall leave that to one side for the moment. There are numerous ways of managing changes to config files, one method (using token replacement) is covered here. The important thing is that this step is carefully managed. I would always prefer to automate this step and write a test script to check that the configs look like I expect them to, rather than ever get involved in manually updating them – the risk of error is simply too high.

What I like to do is store all the configs in source control (I usually use svn), and then either include them in the build (so that when you actually get a release, the configs are all already in there), or get them from the tag branch. Either way, I like them to be tokenised. Then I use the method shown above to replace the tokens with proper values. I like to do this at deploy time because at that point you know the destination hosts you’re going to do deployments to, and can therefore retrieve the right corresponding values to replace the tokens. I also like the practice of keeping the values in a db, and simply pulling them out of the db at deploy time. There are obviously a million ways you can do this. You can even embed it in the ant deploy script like so:

<target name=”getDbName”>

<property name=”temp_sql” value=”${temp.dir}/getdbname.sql”/>

 

<echo file=”${temp_sql}”>select dbname from configuration where servername = ‘${destination.name}’ </echo>

 

<exec executable=”${psql_exec}” failonerror=”false” outputproperty=”dbname.tmp”>

<env key=”PGPASSWORD” value=”${dbpasswd}”/>

<arg line=”-h ${dbsrvname}”/>

<arg line=”-U ${dbusr}”/>

<arg line=”-d ${dbname}”/>

<arg line=”-f ${temp_sql} -t”/>

</exec>

<echo>${dbname}</echo>

</target>

The package is now “ready” to copy over to the destination host – i.e. the configs files are correct for the destination we’re pushing to, the package structure is correct, and any third party files have been included (see jre/jdk above). I tend to tar or zip up the package, simply because it makes the package smaller, and this can be useful if you’re copying the release package to another datacentre and bandwidth is unpredictable. Of course, this step is entirely optional. Anyway, in keeping with the previous examples, here’s an ant snippet:

<target name=”zip”>

<zip destfile=”${release.dir}/release.zip” basedir=”${release.dir}” update=”true”/>

</target>

As simple as that. We’re now ready to move on and automate steps 6-10, which is in the next post!

Best Practices for Build and Release Management Part 2

Ok, as promised in Part 1, I’ll go into a bit more detail about each of the areas outlined previously, starting with…

The Build Process

This area, perhaps more than any other area I’ll be covering in this section, has benefited most from the introduction of some ultra handy tools. Back in the day, building/compiling software was fairly manual, and could only be automated to a certain degree, make files and batch systems were about as good as it got, and even that relied on a LOT of planning and could quite often be a nightmare to manage.

These days though, the build phase is exceedingly well catered for and is now a very simple process, and what’s more, we can now get an awful lot more value out of this single area.

As I mentioned before, one of the aims of release management is to make software builds simple, quick and reliable. Tools such as Ant, Nant (.Net version of Ant), Maven, Rake and MSBuild help us on our path towards our goal in many ways. Ant, MSBuild and Nant are very simple XML based scripting languages which offer a wide ranging level of control – for instance, you can build entire solutions with a single line of script, or you can individually compile each project and specify each dependency – it’s up to you to decide what level of control you need. I believe that build scripts should be kept simple and easy to manage, so when dealing with NAnt and MSBuild for .Net solutions I like to build each project by calling an .proj file rather than specifically compiling each library. The .proj files should be constructed correctly and stored in source control. Each build should get the latest proj file  (and the rest of the code, including shared libraries – more on that later) and compile the project.

For Java projects. Ant and Maven are the most popular tools. Ant, like Nant, gives the user a great deal of control, while Maven has less inherent flexibility and enforces users to adhere to its processes. However, both are equally good at helping us make our build simple, quick and reliable. Maven uses POM files to control how projects are built. Within these POM files a build engineer will define all the goals needed to compile the project. This might sound a little tedious but the situation is made easier by the fact that POM files can inherit from master/parent POM files, reducing the amount of repetition and keeping your project build files smaller, cleaner and easier to manage. I would always recommend storing as much as possible in parent POM files, and as little as you can get away with in the project POMs.

One of the great improvements in software building in recent years has been the introduction of Continuous Integration. The most popular CI tools around are CruiseControl, CruiseControl.Net, Hudson and Bamboo. In their simplest forms, CI tools are basically just schedulers, and they essentially just kick off your build tools. However, that’s just the tip of the iceberg, because these tools can do much, MUCH more than that – I’ll explain more later, but for now I’ll just say that they allow us to do our builds automatically, without the need for any human intervention. CI tools make it very easy for us to setup listeners to poll our source code repositories for any changes, and then automatically kick off a build, and then send us an email to let us know how the build went. It’s very simple stuff indeed.

So let’s take a look at what we’ve done with our build process so far:

  • We’ve moved away from manually building projects and started using simple build scripts, making the build process less onerous and not so open to human error. Reliability is on the up!
  • We’ve made our build scripts as simple as possible – no more 1000 line batch files for us! Our troubleshooting time has been significantly reduced.
  • We’ve moved away from using development UIs to make our builds – our builds are now more streamlined and faster.
  • We’ve introduced a Continuous Integration system to trigger our builds whenever a piece of code is committed – our builds are now automated.

So in summary, we’ve implemented some really simple steps and already our first goal is achieved – we’ve now got simple, quick and reliable builds. Time for a cup of tea!

Best Practices for Build and Release Management Part 1

Firstly, Release Management has been around for long enough for it to no longer mean what it used to mean. Release Management used to be concentrated on the discipline of “creating a release of software”, that generally involved the following key points:

  • How to create or build a reliable “release”
  • How to get that reliable release out into the wild

The sorts of issues that these key points in turn raised were things like:

  • How to reliably and repeatably “build” (compile) software
  • How to make software builds quicker
  • How to make software builds easier
  • How to package software builds (zips, .msi etc)

We used to spend our time working with make files, batch files and countless checklists, running manual builds, and then we’d painstakingly create installers or configure zip files to deploy our releases. And when things went wrong, they usually went seriously wrong, and repeating the build and release process could take days.

Since those bad old days, Release Management has come a long way. Lots of the old issues have been addressed by some exceedingly neat tools which have placed emphasis on automation and quality (I’m thinking Ant/Nant, Cruise Control, the Continuous Integration process, Hudson and loads more). But one other major thing has happened in the world of Release Management, and that’s ITIL.

ITIL has redefined the practice of Release Management as more of a planning and coordinating role, it even goes so far as to say Release Management involves communicating with customers and managing customer expectation. This is a million miles away from writing complex batch files, hundreds of lines long, to compile and deploy software to a QA environment! In an ITIL world, the issues listed earlier either don’t exist, or have been addressed already and are no longer a concern to a Release Manager.

So why does the ITIL version of Release Management differ so much from the real world job of a Release Manager?

Well, I would guess that the “build management” aspect is simply not considered part of release management, and that it should be covered somewhere else, but that’s just my guess, I’m seeking some advice from ITIL about that right now.

What we’re left with now is a world where “Release Management” means one thing to one person, and something completely different to another. I’m from the old school of Release Management, I like to actually produce stuff. In a second I’ll outline what I consider to be the main roles and objectives of Release Management, and then later I’ll take each one and explain some ways that I’ve used for tackling them.

So, I like to think of Release Management as a practice which:

  • Helps make software builds simple, quick and reliable. This is achieved by employing the best tools for the job. This means understanding all the various build tools, seeing how they integrate with the systems that already exist in the workplace, and making an informed choice. There’s no way you’re going to make software builds easier, more reliable and repeatable by implementing a manual solution, so get to grips with the various build tools out there and make them work for you.
  • Helps make software deployments simple, quick, reliable and repeatable. Again, this is a bit like the above, but there are fewer tools to choose from. Manually deploying releases is painful and risky, and it also belongs in the dark ages and should be outlawed. There are still plenty of options and combinations of tools to make this task fully automated.
  • Helps take care of configuration management. When I say configuration management, I’m talking about all those issues with how to make a software release look, feel and behave the same from one environment to the next. For me this falls into Release Management because Release Management, unlike development, QA or Operations, has a direct involvement in every environment along the way to releasing into the wild. It’s pointless asking the development team to tackle the issues of configurations between environments when they have very little or no visibility of the production environment, and besides, their time would be much better spent making that button look cooler because that’s what the business has asked for!
  • Helps drive software quality. Thanks to the Continuous Integration process, and the tools that have been built around it, it’s now possible for us to build software every single time a piece of code is checked in, run a suite of unit tests, analyse the code for lazy programming and report on the amount of test coverage a project has. And that’s just the start. There are tools out there for doing much much more than this, and I’ll go into more detail about this later.
  • Helps optimise development and QA time. By giving the dev team the feedback on the quality of their code and telling them where they’re going right and going wrong, we’re helping them target their efforts. Furthermore, if were busy providing these solutions for them, doing the builds, configurations and releases, the developers can get busy doing the stuff they’re skilled at doing. For the QA team, we’re finding bugs and failing releases before the releases even get to them! (of course, if we find too many bugs and fail a release, that release won’t even get o QA)
  • Speeds up time to market. Ok, so we’ve made builds quicker, easier and more reliable, we’ve sped up the process of fine tuning code quality, we’ve spotted bugs before a round of QA has even begun and we’ve made the process of releasing our software out into the wild quicker and simpler. Basically we’ve saved a heap of time in dev, QA and Operations and so our new, higher quality software, can be released efficiently into the wild. Happy days!

As promised earlier, I’ll spend a while giving a few examples of how to actually implement what I’ve broadly outlined above. I’ll try to be generic where I can, but I’ll include specifics for some examples. All that and more in Part 2!