Uncategorized | DevOpsNet

Build Reports For the Project Team

Posted on November 10, 2010 by jamesbetteley

Reading through Jez Humble and David Farley’s new(ish) book “Continuous Delivery” at the moment. It’s just great to read a book where the authors are really speaking your language. The concepts are simple and correct. There are aslo a few new ideas in there that I’ve not thought of before. But most of all I like the way it gets me thinking about how to implement some of the concepts in a “challenging” atmosphere. Not all companies understand the value of Continuous Integration, let alone Continuous Deployment/Delivery, but this book lays out the concepts in such a simple, straightforward way that it seems hard to believe that anyone would not see the advantages in them.

I picked up on one idea which I’d like to start implementing ASAP – and that is to provide feedback to project teams about the state of the builds and environments involved in that project, throughout the whole project lifecycle. I think that might help the rest of the project team understand the importance of delivery, by drawing attention to and raising awareness of the builds and the deployment environments. I can well imagine that some project team members won’t even know what they’re looking at when they’re presented with a build report, but there’s no reason for this. Everyone on the project should be interested in the build reports, they should have a vested interest in the quality of the code and the state of the deployment environment. These are real, tangible things, not abstract concepts, so they should be presented as real, tangible things for everyone to see.

Best Practices for Build and Release Management Part 3: Making software deployments rapid, reliable and repeatable

Posted on November 10, 2010 by jamesbetteley

When we do our deployments, it’s usually at that time in the project everyone in the world is eager for something to be delivered. With all eyes on you, the last thing you want is a complicated, arduous, risk-filled deployment task. What you need is a rapid, reliable and repeatable task that you have the utmost faith in.

Of course, the simple solution in a nutshell is to automate deployments. This easier said than done of course, but it’s also not rocket science.

If I think of some of the deployments I’ve done in the past and break them down into their constituent manual steps, I might come up with something along these lines:

Download a tar ball from a repository
Extract the tar and move some files.
Make changes to the config files
Add some third party files (maybe tomcat for instance)
Tar it back up
Prep target servers by removing the existing version
Send the new version to each server
Extract it on each host
Start the application
Test that it works

That’s quite a few steps, and in reality each one of these steps would probably consist of several others.

These tasks might not seem risk-filled and arduous but in actual fact they are. If all these steps were manual, then there’d be ample opportunity for human error. Perhaps the downloading of the tar file was incomplete – we’ve introduced a machine error that might not be caught until step 10!

What about the configurations? We might have different configurations for each locale. In a manual process we might have to do this by hand. The opportunity for human error is greatly extended, and mistakes in config files can be costly.

And what about the sheer amount of time it would take if we tried to do these tasks ourselves? I know I’d rather be getting on with something else more constructive!

So let’s see what we can do about making this deployment rapid, reliable and repeatable.

In actual fact we can easily automate all these steps.

We can write a script to download the tar file and then check it against an md5 checksum. An Ant script could do this for us nicely, and hey presto, we’ve removed the risk of that machine error.

Here’s an Ant code snippet for getting a tar file from a maven repository (handy if you use Maven ;-)) and doing an md5 check:

<target name=”get_tar”>
<echo>Getting the distribution from ${sourcehost}</echo>
<scp file=”${sourcefile}” todir=”${tars.dir}/${app.name}/${filename}” trust=”true” keyfile=”${key}” passphrase=” “/>
</target>

Next Step – extracting the file and moving some files about: Really we should look to minimise the amount of file moving we need to do, if the application doesn’t get delivered in the right structure then this should be addressed earlier in the process so that it does get delivered in exactly the right structure for our production system. Go back to the developer/vendor and tell them how you would expect the delivery to look. If this isn’t an option, I’ve again used ant tasks to get my releases into the state I want them in. Here’s an example:

<!– copy the tomcat files into the release –>

</fileset>

</copy>

<!– copy the application configs files into the right directory –>

</filterset>

</copy>

As you can see, I’ve also made some changes to the configs files here while I copied them from one directory to another. In theory, the only difference between an application deployed on a test environment, and the same application deployed on the production environment, should be the config files. In reality we also have databases which can affect the functionality of our system, but we shall leave that to one side for the moment. There are numerous ways of managing changes to config files, one method (using token replacement) is covered here. The important thing is that this step is carefully managed. I would always prefer to automate this step and write a test script to check that the configs look like I expect them to, rather than ever get involved in manually updating them – the risk of error is simply too high.

What I like to do is store all the configs in source control (I usually use svn), and then either include them in the build (so that when you actually get a release, the configs are all already in there), or get them from the tag branch. Either way, I like them to be tokenised. Then I use the method shown above to replace the tokens with proper values. I like to do this at deploy time because at that point you know the destination hosts you’re going to do deployments to, and can therefore retrieve the right corresponding values to replace the tokens. I also like the practice of keeping the values in a db, and simply pulling them out of the db at deploy time. There are obviously a million ways you can do this. You can even embed it in the ant deploy script like so:

<echo file=”${temp_sql}”>select dbname from configuration where servername = ‘${destination.name}’ </echo>

</exec>

<echo>${dbname}</echo>

</target>

The package is now “ready” to copy over to the destination host – i.e. the configs files are correct for the destination we’re pushing to, the package structure is correct, and any third party files have been included (see jre/jdk above). I tend to tar or zip up the package, simply because it makes the package smaller, and this can be useful if you’re copying the release package to another datacentre and bandwidth is unpredictable. Of course, this step is entirely optional. Anyway, in keeping with the previous examples, here’s an ant snippet:

</target>

As simple as that. We’re now ready to move on and automate steps 6-10, which is in the next post!

Fun with FindBugs

Posted on November 3, 2010 by jamesbetteley

We’ve just moved to a new master pom file in an effort to make our lives a bit easier, and to allow certain builds to carry on using the old master pom file which was basically quite rubbish. You see, the old master pom file just defined a load of plugins, mostly in the plugin management section, so they had to be referenced by the application poms anyway. The idea with the new master pom is that it enforces the use of certain plugins, and ALSO enfiorces certain standards for builds to pass – for instance, we included a cobertura coverage rate of 80% and made the builds fail if there were any findbugs issues. It sounded like a good idea at the time. So, we put these plugin definitions directly into the build section of the pom, like so:

<build>

…

<plugins>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>

<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.4</version>
<configuration>
<testClassesDirectory>
build/maven/${artifactId}/target/test-classes
</testClassesDirectory>
</configuration>
</plugin>

<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<version>2.4</version>
<configuration>
<formats>
<format>html</format>
<format>xml</format>
</formats>
<check>
<totalBranchRate>54</totalBranchRate>
<totalLineRate>75</totalLineRate>
</check>
</configuration>
<executions>
<execution>
<goals>
<goal>clean</goal>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-pmd-plugin</artifactId>
<version>2.5</version>
<executions>
<execution>
<goals>
<goal>check</goal>
<goal>cpd-check</goal>
</goals>
</execution>
</executions>
<configuration>
<linkXref>true</linkXref>
<targetJdk>1.6</targetJdk>
<sourceEncoding>utf-8</sourceEncoding>
<failOnViolation>false</failOnViolation>
<outputDirectory>build/maven/${pom.artifactId}/target/pmd-reports</outputDirectory>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>findbugs-maven-plugin</artifactId>
<version>2.3.1</version>
<!– NOT USING THIS YET
<executions>
<execution>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions> –>
</plugin>
</plugins>
</build>

The first thing I had to do (as you can see) is disable the check goal because it was causing findbugs to fail pretty much every build. The next thing I had to do was remove the whole of the following configuration section from the findbugs plugin:

<configuration>
<threshold>High</threshold>
<effort>Max</effort>
<xmlOutput>true</xmlOutput>
<xmlOutputDirectory>build/maven/${artifactId}/target/site</xmlOutputDirectory>
</configuration>

I’ve kept this section in the reporting section though.

The next thing I got was this error:

03-Nov-2010 09:53:03	[java] Java Result: 1
03-Nov-2010 09:53:03	[Fatal Error] findbugsTemp.xml:1:1: Premature end of file.
03-Nov-2010 09:53:03	[INFO] ————————————————————————
03-Nov-2010 09:53:03	[ERROR] FATAL ERROR
03-Nov-2010 09:53:03	[INFO] ————————————————————————
03-Nov-2010 09:53:03	[INFO] Premature end of file.
03-Nov-2010 09:53:03	[INFO] ————————————————————————
03-Nov-2010 09:53:03	[INFO] Trace
03-Nov-2010 09:53:03	org.xml.sax.SAXParseException: Premature end of file.

and that was fixed by changing this bit of the findbugs plugin configuration:

to this:

<effort>Default</effort>

So, in short, using “Max” gives us java out of memory exceptions. Which is not exactly convenient. Not sure how to fix this though.

Bamboo error: build cannot be preformed since you have local modifications

Posted on September 15, 2010 by jamesbetteley

This happens a lot. Thankfully it’s easy to fix:

Basically you just have to add the svn:ignore on build-number.txt

1. In TortoiseSVN select “Properties”

2. Click the “Add” button

3. Define the svn:ignore property:

4. COMMIT THE CHANGES!!! Don’t forget to do this or you’ll keep getting the error.

Automate Configuration Management Using Tokens!

Posted on August 9, 2010 by jamesbetteley

Here’s the problem:

Your application has numerous config files, and the values in these config files differ on every server or every environment. You hate manually updating the values every time you deploy your applications to a new environment, because that takes up too much of your time and inevitably leads to costly mistakes.

Here’s the solution:

Automate it.

And here’s one way of doing it:

Use “master” config files that have ALL environmental details replaced with tokens
Move copies of these files to folders denoting the environments they’ll be deployed to
Use a token replacement operation to replace the tokens
Deploy over the top of your code deployments, in doing so replacing the default config files

All the above can be automated very easily, and here’s how:
First off, make tokenised copies of your config files, so that environmental values are replaced with tokens, e.g.
change things like:

<add key=”DB:Connection” value=”Server=TestServer;Initial Catalog=TestDB;User id=Adminuser;password=pa55w0rd”/ >
to

Then save a copy of these tokens, and their associated values in a sed file. This sed file should contain values specific to one environment, so that you’ll end up with 1 sed file per environment. These files act as lookups for the tokens and their values.

The sybntax for these sed files is:

s/%TOKEN%/TokenValue/i

So here’s the contents of a test environmemt sed file (testing.sed)

s/%DB_SERVER%/TestServer/i

s/%DB_NAME%/TestDB/i

s/%DB_UID%/Adminuser/i

s/%DB_PWD%/pa55w0rd/i

And here’s live.sed:

s/%DB_SERVER%/LiveServer/i

s/%DB_NAME%/LiveDB/i

s/%DB_UID%/Adminuser/i

s/%DB_PWD%/Livepa55w0rd/i

Next up, we want to have a section in our build script which renames the web_master.config files and copies them, and then runs the token replacement task….so here it is:

</fileset>

</copy>

</fileset>

</copy>

</target>

</target>

As you can see, the last target calls a couple of cmd files, the first of which looks like this:

xfind “%*\TestArea” -iname *.* xargs sed -i -f “D:\compiled\config\testing.sed”

xfind “%*\TestArea” -iname *.* xargs sed -i s/$/\r/

This is the sed command to read the config file, pipe the contents to sed and run the script file against it, and edit it in place. the second line handles Line Feeds so that the file ends up in a readable state. Essentially we’re telling sed to recursively read through the config file, and replace the tokens with the relevant value.

The advantage that this method has over using Nant’s “replacetokens” is that we can call the script for any number of files in any number of subdirectories using just one call, and the fact that the tokens and values are extracted from the build script. Also, the syntax means that the sed files are a lot smaller than a similar functioning Nant script would be.

Of course, you could make this whole thing even more elegant by putting the token/value pairs in a database instead of in a sed file, simply pull them out of the db at build/deploy time and then do the substitution.

People sometimes say that this method doesn’t work well if there are a large number of config files; they don’t like the idea of maintaining a large number of “master” versions as well as standard code versions. So to get around this, you can just not use tokens, but instead have the sed/replace look for the xml node and then the element, and simply replace the value there. There are plenty of ways of doing this using xml xPath. Both approaches have their own advantages, I guess the decision of which one to go for could just be a matter of how numerous your config files are.

Best Practices for Build and Release Management Part 2

Posted on July 22, 2010 by jamesbetteley

Ok, as promised in Part 1, I’ll go into a bit more detail about each of the areas outlined previously, starting with…

The Build Process

This area, perhaps more than any other area I’ll be covering in this section, has benefited most from the introduction of some ultra handy tools. Back in the day, building/compiling software was fairly manual, and could only be automated to a certain degree, make files and batch systems were about as good as it got, and even that relied on a LOT of planning and could quite often be a nightmare to manage.

These days though, the build phase is exceedingly well catered for and is now a very simple process, and what’s more, we can now get an awful lot more value out of this single area.

As I mentioned before, one of the aims of release management is to make software builds simple, quick and reliable. Tools such as Ant, Nant (.Net version of Ant), Maven, Rake and MSBuild help us on our path towards our goal in many ways. Ant, MSBuild and Nant are very simple XML based scripting languages which offer a wide ranging level of control – for instance, you can build entire solutions with a single line of script, or you can individually compile each project and specify each dependency – it’s up to you to decide what level of control you need. I believe that build scripts should be kept simple and easy to manage, so when dealing with NAnt and MSBuild for .Net solutions I like to build each project by calling an .proj file rather than specifically compiling each library. The .proj files should be constructed correctly and stored in source control. Each build should get the latest proj file (and the rest of the code, including shared libraries – more on that later) and compile the project.

For Java projects. Ant and Maven are the most popular tools. Ant, like Nant, gives the user a great deal of control, while Maven has less inherent flexibility and enforces users to adhere to its processes. However, both are equally good at helping us make our build simple, quick and reliable. Maven uses POM files to control how projects are built. Within these POM files a build engineer will define all the goals needed to compile the project. This might sound a little tedious but the situation is made easier by the fact that POM files can inherit from master/parent POM files, reducing the amount of repetition and keeping your project build files smaller, cleaner and easier to manage. I would always recommend storing as much as possible in parent POM files, and as little as you can get away with in the project POMs.

One of the great improvements in software building in recent years has been the introduction of Continuous Integration. The most popular CI tools around are CruiseControl, CruiseControl.Net, Hudson and Bamboo. In their simplest forms, CI tools are basically just schedulers, and they essentially just kick off your build tools. However, that’s just the tip of the iceberg, because these tools can do much, MUCH more than that – I’ll explain more later, but for now I’ll just say that they allow us to do our builds automatically, without the need for any human intervention. CI tools make it very easy for us to setup listeners to poll our source code repositories for any changes, and then automatically kick off a build, and then send us an email to let us know how the build went. It’s very simple stuff indeed.

So let’s take a look at what we’ve done with our build process so far:

We’ve moved away from manually building projects and started using simple build scripts, making the build process less onerous and not so open to human error. Reliability is on the up!
We’ve made our build scripts as simple as possible – no more 1000 line batch files for us! Our troubleshooting time has been significantly reduced.
We’ve moved away from using development UIs to make our builds – our builds are now more streamlined and faster.
We’ve introduced a Continuous Integration system to trigger our builds whenever a piece of code is committed – our builds are now automated.

So in summary, we’ve implemented some really simple steps and already our first goal is achieved – we’ve now got simple, quick and reliable builds. Time for a cup of tea!

Best Practices for Build and Release Management Part 1

Posted on July 21, 2010 by jamesbetteley

Firstly, Release Management has been around for long enough for it to no longer mean what it used to mean. Release Management used to be concentrated on the discipline of “creating a release of software”, that generally involved the following key points:

How to create or build a reliable “release”
How to get that reliable release out into the wild

The sorts of issues that these key points in turn raised were things like:

How to reliably and repeatably “build” (compile) software
How to make software builds quicker
How to make software builds easier
How to package software builds (zips, .msi etc)

We used to spend our time working with make files, batch files and countless checklists, running manual builds, and then we’d painstakingly create installers or configure zip files to deploy our releases. And when things went wrong, they usually went seriously wrong, and repeating the build and release process could take days.

Since those bad old days, Release Management has come a long way. Lots of the old issues have been addressed by some exceedingly neat tools which have placed emphasis on automation and quality (I’m thinking Ant/Nant, Cruise Control, the Continuous Integration process, Hudson and loads more). But one other major thing has happened in the world of Release Management, and that’s ITIL.

ITIL has redefined the practice of Release Management as more of a planning and coordinating role, it even goes so far as to say Release Management involves communicating with customers and managing customer expectation. This is a million miles away from writing complex batch files, hundreds of lines long, to compile and deploy software to a QA environment! In an ITIL world, the issues listed earlier either don’t exist, or have been addressed already and are no longer a concern to a Release Manager.

So why does the ITIL version of Release Management differ so much from the real world job of a Release Manager?

Well, I would guess that the “build management” aspect is simply not considered part of release management, and that it should be covered somewhere else, but that’s just my guess, I’m seeking some advice from ITIL about that right now.

What we’re left with now is a world where “Release Management” means one thing to one person, and something completely different to another. I’m from the old school of Release Management, I like to actually produce stuff. In a second I’ll outline what I consider to be the main roles and objectives of Release Management, and then later I’ll take each one and explain some ways that I’ve used for tackling them.

So, I like to think of Release Management as a practice which:

Helps make software builds simple, quick and reliable. This is achieved by employing the best tools for the job. This means understanding all the various build tools, seeing how they integrate with the systems that already exist in the workplace, and making an informed choice. There’s no way you’re going to make software builds easier, more reliable and repeatable by implementing a manual solution, so get to grips with the various build tools out there and make them work for you.
Helps make software deployments simple, quick, reliable and repeatable. Again, this is a bit like the above, but there are fewer tools to choose from. Manually deploying releases is painful and risky, and it also belongs in the dark ages and should be outlawed. There are still plenty of options and combinations of tools to make this task fully automated.
Helps take care of configuration management. When I say configuration management, I’m talking about all those issues with how to make a software release look, feel and behave the same from one environment to the next. For me this falls into Release Management because Release Management, unlike development, QA or Operations, has a direct involvement in every environment along the way to releasing into the wild. It’s pointless asking the development team to tackle the issues of configurations between environments when they have very little or no visibility of the production environment, and besides, their time would be much better spent making that button look cooler because that’s what the business has asked for!
Helps drive software quality. Thanks to the Continuous Integration process, and the tools that have been built around it, it’s now possible for us to build software every single time a piece of code is checked in, run a suite of unit tests, analyse the code for lazy programming and report on the amount of test coverage a project has. And that’s just the start. There are tools out there for doing much much more than this, and I’ll go into more detail about this later.
Helps optimise development and QA time. By giving the dev team the feedback on the quality of their code and telling them where they’re going right and going wrong, we’re helping them target their efforts. Furthermore, if were busy providing these solutions for them, doing the builds, configurations and releases, the developers can get busy doing the stuff they’re skilled at doing. For the QA team, we’re finding bugs and failing releases before the releases even get to them! (of course, if we find too many bugs and fail a release, that release won’t even get o QA)
Speeds up time to market. Ok, so we’ve made builds quicker, easier and more reliable, we’ve sped up the process of fine tuning code quality, we’ve spotted bugs before a round of QA has even begun and we’ve made the process of releasing our software out into the wild quicker and simpler. Basically we’ve saved a heap of time in dev, QA and Operations and so our new, higher quality software, can be released efficiently into the wild. Happy days!

As promised earlier, I’ll spend a while giving a few examples of how to actually implement what I’ve broadly outlined above. I’ll try to be generic where I can, but I’ll include specifics for some examples. All that and more in Part 2!