Upcoming DevOps & Agile Events

London Puppet User Group Meetup
London, Thursday May 21st, 2015
6:00pm
http://goo.gl/C2zuKb

DevOps Exchange London – DevOps & DevOps
London, Tuesday May 26th, 2015
6:30pm
http://goo.gl/Xmdqxl

London Agile Discussion Group – Should DevOps be a person or a team-wide skill?
London, Tuesday May 26th, 2015
6:30pm
http://goo.gl/xksVOH

AWS User Group UK – meetup #15
London, Wed May 27th, 2015
6:30pm
http://goo.gl/uBsiUj

Chef Users London – Microsoft Azure / Chef Taster Day
London, Friday May 29, 2015
9:00am to 5:00pm
http://goo.gl/VOvkC3

DevOps Cardiff – Herding ELKs with consul.io
Cardiff, Wednesday, June 3, 2015
6:30pm
http://goo.gl/WwOvkQ

Agile Testing – Visual Creativity: Using Sketchnotes & Mindmaps to aid testing @ #ltgworkshops
London, Thursday June 4th, 2015
8:30am
http://goo.gl/34iIXM

ABC (Agile Book Club) London – Review Jeff Patton’s User Story Mapping
London, Thursday June 4th, 2015
6:30pm
http://goo.gl/X0qPwb

Agile Testing – Hooking Docker Into Selenium @ #ltgworkshops
London, Thursday June 4th, 2015
8:30am
http://goo.gl/ONH8dQ

UK Azure User Group – Cloud Gaming Hackathon
London, Saturday June 6th, 2015
9:30am
http://goo.gl/ONH8dQ

London DevOps – London DevOps Meetup #10
London, Thursday June 11th, 2015
7:00pm
http://goo.gl/uolxJk

Kanban Coaching Exchange – Continuous learning through communities of practice – Emily Webber
London, Thursday June 11th, 2015
6:30pm
http://goo.gl/9aFD8x

Lean Agile Manchester
Manchester, Wednesday June 17th, 2015
6:30pm
http://goo.gl/Z15ac3

London Lean Coffee – Holborn
London, Thursday, June 18th, 2015
9-10am
http://goo.gl/QkIBhj

UK Azure User Group – Chris Risner
London, Thursday June 18th, 2015
7:00pm
http://goo.gl/EfbNnn

Jenkins User Conference – Europe (London)
London, Tuesday June 23rd – 24th, 2015
2 days
http://goo.gl/achJJX

BDD London June Meetup
London, Thursday June 25th, 2015
6:30pm
http://goo.gl/C2zuKb

Automated Database Deployment (Workshop – £300)
Belfast, Northern Ireland, Friday June 26th, 2015
1 day course
http://goo.gl/fXlJr7

Database Continuous Integration (Workshop – £300)
London, July 8th, 2015
1 day course
http://goo.gl/lW4TjA

Database Source Control (Workshop – £100)
London, July 8th, 2015
1 day course
http://goo.gl/C2zuKb

London Lean Coffee – Holborn
London, Thursday, July 16, 2015
9-10am
http://goo.gl/mtJ3k4

Agile Taster – a free introductory Agile training course
Cardiff, Saturday 18 July 2015
10am – 3pm
http://goo.gl/qFYS6b

AWS User Group UK – meetup #16
London, Wed July 22nd, 2015
6:30pm
http://goo.gl/Tc3hlD

Advertisement

Continuous Improvement – 10 Ways to Help Your Team Learn (plus 6 more)

Not long ago I went to one of the Agile Coaching Exchange’s meetups in the lovely asos offices in London. Speaker for the night was none other than Rachel Davies who I worked with about a decade ago when she was a freelance agile coach. My god that decade has gone quickly. Anyway, her talk was about the techniques that they use at unruly to encourage learning in the workplace, and as you’d expect, it was really interesting stuff. So, I decided to take some notes and even give some of her ideas a go. Here’s what happened:

Learning Techniques

At one point Rachel asked us, the unsuspecting audience, to come up with a list of different learning techniques we’ve used in the workplace. It was a trap. No matter how many I thought we’d covered off, we were nowhere near the list that Rachel came up with. Basically we’re just not as cool as those kids over at Unruly, that’s what I learned. Anyway, keen to learn more about learning (woah, Learning Inception!) I decided to list the learning techniques I liked the sound of, and I’ve added a bunch of others that hopefully you’ll like the sound of as well (because, you know, lists are way cool):

  1. Workshops
  2. Attending meetups
  3. Pairing
  4. Retrospectives
  5. Mobbing
  6. Hackdays
  7. Devdays
  8. 20% Time
  9. Tech Talks
  10. Book clubs
  11. Coding Dojos
  12. Team Swaps
  13. Rotation
  14. Tech Academy
  15. Blogs
  16. Conferences

Workshops – I use these a lot in my work. I mostly try to keep them hands-on, encouraging the attendees to physically get involved. If I was any good at marketing I would probably describe them as “Interactive”. If necessary, I’ll use hand-outs, but I’ll never just stand there talking through a bunch of slides – that’s seriously uncool and you’ll never get into the Secret Inner Sanctum of the Workshop Magic Circle if you do that. The objective is for the attendees to be actively involved in the workshop, rather than to simply be an observer. I run workshops on Agile Product Ownership, Kanban, Flow (Theory of Constraints), and Sprint Planning & Estimating. Remember, using the term “Workshop” isn’t just a way of making a 4 hour meeting sound more interesting 🙂

Attending Meetups is not only a great way of learning from whatever the speakers are talking about, but also from chatting with the other people at the meetup. I regularly attend the London Continuous Delivery Meetup group (where you get the chance to pick the brains of people such as Matthew Skelton, Steve Smith and Chris O’Dell), the London Devops Exchange, the London Devops Meetup (where you can casually run your devops problems by Marc Cluet and Matt Saunders and then listen while they give you a solution) and the Cardiff DevOps Meetup (hosted by the DevOpsGuys, so you can be guaranteed some top-notch speakers as well as the best beer in the business – I kid you not, at devopsguys we have our own beer!)

Yep, it's called DevHops

Yep, it’s called DevHops

Pairing – Like other programmers of my particular skill level (pisspoor), I get very self-conscious whenever I’m pairing. Not only when I’m the one driving, but also when I’m observing, because I ask stooooopid questions. Below is a picture of me pair programming with my son, who, despite being unable to speak yet, is clearly getting annoyed at my stupid questions (I think I just asked him what nested ternary operators are). However, there’s no denying it’s a fantastic way of learning. The technique we’re trying below involves me writing some ruby function, and then my son will refactor it and embarrass me.

IMG_20150329_114207[1]

 

Retrospectives are a way of reflecting on your latest sprint or release, and talking about what you did well, as well as what you didn’t do so well. The trouble is though, that you have to actually take these lessons on-board, and start implementing changes if necessary. It’s all very well reflecting on your performance, but it won’t improve unless you actually do something about it. This could be a whole blog post of its own, bust basically I’m seeing a lot of people in this situation where they rigorously do retrospectives, but nobody every implements the lessons learnt. Quite often it’s because there’s no agile coach involved with the team (and without the agile coach, nobody else has the time to implement the relevant changes themselves, let alone feels responsible for doing so).

Mobbing – Another picture coming up. This time another member of my family is joining in a mobbing session, which is basically a bunch of people all working on the same problem simultaneously (usually around the same screen). Like pairing, it’s a great learning technique. In fact I think it’s superior to pairing, because there are more people and therefore more minds on the job. But of course it can be costly to tie up multiple people on the same task.

Mobbing with my son and Tygwydd

Mobbing with my son and Tygwydd

Hackdays are like a geek-off for devs. I once spent the first 4.5 hours of a hackday trying to install LAMP before basically throwing my PC out of a window. Hackdays are where you get a bunch of devs together and give them all a problem to solve, or some objective to reach (you can be a specific or as vague as you like – often the more vague you are, the more creative your devs will be). 24hrs and a lot of pizza later, you’ll have a bunch of interesting creations – some more complete than others, but all of them creative, geeky, and in their own way very cool. I guarantee you’ll never see a passionate software developer work harder than during a hackday. What do you learn from a hackday? As a dev, you learn how to concentrate after 8 cans of Red Bull, and if you’re in a team then you learn how to work as a team under a high-pressure environment.

Devdays are something I really like to encourage within my teams. The idea is that for at least one day a sprint, 1 or 2 of your delivery team can work on something outside of the sprint commitments. I would aim to make sure everyone gets to take a devday at least once every 3 sprints. Of course, it needs to be relevant work, and it needs to be scheduled ahead of time (get into the habit of asking if anyone’s planning on taking a devday during sprint planning). If your team aren’t doing devdays, it’s a sure sign that you’re either too busy (and will end up experiencing burn-out) or your devs are disinterested. Devdays are a great opportunity to learn a new tool or to start spiking a new idea, perhaps using a new language.

20% Time is fairly similar to the devdays concept, in that people are encouraged to spend up to 1 day a week working on something that’s not on the backlog. I think the idea came from Google, but I’m not sure if they still practice it. Basically devdays, gold-card days or 20% time, call it what you will, are all designed to encourage learning and innovation and keep people feeling fresh and engaged. During her talk, Rachel spoke a little about Gold Cards, which I’d love to tell you more about, but I had to go and take a call just as she was talking about them, so you’ll just have to go and read more about them here.

Tech Talks are like little mini meetups, usually within an organisation, but companies like Facebook also do public tech-talks as well. Great for learning and eating free pizza and doughnuts. As a general rule, if there are no free nibbles, don’t go. Facebook had exceptionally good nibbles at their tech-talk. Just like at meetups, they’re a great place for tapping into the brain power of your fellow attendees as well as the speaker/presenter.

Book Clubs are one of the most underrated and under-used tools for learning, in my opinion. I ran a book club last year in an organisation that was trying to transition to Agile. The book I chose was called The Agile Samurai by Jonathan Rasmusson, which was a big hit with everyone who joined in. The format I use is for the group to read a couple of chapters of a book over the course of a week, and then have a review session where we all discuss what we’ve learnt. It’s a great way to share what we’ve learnt (which helps to make sure we’re all on the same page) and it also ensures that everyone is progressing at a reasonable pace.

sam

Coding Dojos – These are coding-centric programming clubs, basically. They involve a bunch of eager coders getting together and working (usually on their own laptops or in pairs) on a particular challenge, with the purpose of learning more about a particular language (Ruby, Go, Erlang etc) or technique (BDD, TDD etc). Suffice to say you usually need to have a reasonable amount of programming experience to be able to get the most value out of these, but don’t let that put you off. There are plenty of coding dojo metups available to cater for most levels, or you could of course run one yourself within your own organisation.

Team Swaps are where one team swaps with another for an entire day, or possibly longer. The idea behind this is that if you’re going to hand your codebase over to an entirely different team (and not be around to help), then it teaches you to write clean, self-documenting, simple code. On top of that, it also helps you learn more about other team’s coding styles and techniques.

Rotation – If I had to pick one concept and make it a mandatory part of software development, I would pick rotation. Here’s how it works: you take Danny the developer and put him in QA for a couple of sprints. Meanwhile, you take Tammy the Tester and put her in Dev for a couple of sprints. At a later date, Danny the dev will have to do a stint in the helpdesk, while Tammy does a couple of sprints working with the BA or Product Owner. Until eventually, everyone in your sprint team will have done stints in each of the following teams: Dev, Test, Helpdesk, Infrastructure/ops, Architecture, Product (Product Management, BA or whatever you have in your org), and possibly even Sales. It can take up to a year to complete the full set, but the amount you learn is invaluable. It’s not just skills that you pick up, but most of all it’s the different perspectives you get to see. Eventually, this experience will make you a better software delivery professional.

Tech Academies are becoming quite popular, and we’re seeing an increasing demand for help in setting these up within organisations. The idea is to create a number of internal training courses, tailor-made for the challenges that are unique to your organisation. These could be anything from Agile Coaching courses to Database Administration courses (and everything in between). It’s even quite common to see organisation-specific “certification” as well. People can enrol in one of these academies by choice, or you can make them mandatory, it’s up to you – but the key thing is to make them specific to your organisation’s needs. I think these are exceedingly valuable, and they have the added advantage over external training courses of always being 100% relevant, plus you can also ensure that everyone is getting the same standard of training!

Blogs are a great source of information, and a great way to keep up to date with fellow professionals in your technical area. But don’t just read them, write one for yourself! Keeping a team journal or a company blog is a great way of promoting the cool stuff you’re doing, and is also a great way to encourage and develop people’s technical writing skills (not to mention their written communication skills).

Conferences are a great source of free T-shirts, pens, hats, stress-balls, stickers, key-rings, laser-pointers and other things that you quickly get bored of and leave on your desk at the office. But did you know that you can actually learn stuff at conferences as well? It’s true! Some conferences have really, really clever people speaking at them, (other conferences have me), and you’ll usually find the speakers are more than happy to have a chat with you over a drink after their talk. In all seriousness, the Pipeline conference this year was brilliant – a great crowd of very smart professionals from all walks of life, an inspiring keynote from Linda Rising, and a chilled atmosphere. So, get along to a conference (even if you have to take a devday to get away with it), write down what you learn, make a blog out of it, do a tech-talk to your team about it, expand that into a workshop, maybe include some pairing and/or mobbing, and then head on out to a meetup to chat to more like-minded professionals. 🙂 Learning Level: Einstein!

Adopting Agile in 3 “Easy” Steps

All good plans come in 3 phases:

Profit

Although I won’t be collecting any underpants, I’ll be following this basic template (with a couple of tweaks here and there) during an Agile adoption initiative I’m currently working on.

In the South Park episode (from which I have taken the picture above) the boys discover a bunch of underpant-stealing gnomes, who are collecting underpants as part of a grand plan to make profit. The gnomes claim to be business experts but none of them appears to know what phase 2 of the plan is. All they know is that their business model is based on collecting underpants, and so that’s what they’ll do.

Unfortunately, I have been witness to a couple of attempts at adopting agile which weren’t very dissimilar to the underpants gnomes’ business plan. Namely, a business starts “Adopting Agile”, usually driven by the development team, where they start doing stand-ups and using a sprint board (this is phase 1) and somehow they are surprised when this doesn’t suddenly start producing profit. Clearly, “becoming Agile” isn’t as simple as that.

Phase 1 – Collect business reasons (not underpants)

So you’re going Agile. Presumably you’ve determined that this is what you want, and what your customers need. If you haven’t done this yet then stop right there and ask yourself “Do I Need Go Agile?”. The answer might be “no”, but does needing to go Agile have to be the only reason? Maybe you just want to go agile to see what the fuss is all about, or to make your business more attractive to potential new employees.

brush

So lets assume we’re going agile, and you have valid business reasons to do so. My first suggestion would be to make those business reasons highly visible. You have to outline the existing issues and how Agile can help to fix them. Mitchell and Webb once did a sketch about a toothbrush company who had to try to think of some gimmick to add to their toothbrushes in order to keep increasing their sales. They came up with the idea of “dirty tongue”. This is where microscopic “tonguanoids” build up, and basically result in social exclusion and a lack of sex. Their solution: to put bristles on the other side of the toothbrush so that people can brush their tongues while they brush their teeth. People will buy these toothbrushes despite the fact that “brushing your tongue makes you retch, everybody knows that”. The point I’m making, very badly, is that it’s a lot easier to sell things if people think that what they’re buying into will fix some very real, tangible issue.

The same goes with Agile. To get the buy-in you need to make your agile adoption a success, you’ll need to identify how “going Agile” is going to make life better for everyone concerned.

If the problem you’ve got is that you never ship software on time, or you constantly fail to deliver what the customer wants, then it’s fairly easy to “sell” agile as the solution. The concept of sprints are a doddle for everyone to understand, and they’ll love the idea that the customer will have regular interactions with the development team, and get to see regular progress in the demos. “Of course!” they’ll say “It’s so obvious, why didn’t I think of that before”. The business should easily be able to see how short, sharp sprints with an emphasis on “working software” will make it easier to deliver what the customer wants, and manage their expectations of when it’ll be ready.

But what if those aren’t the problems you need to solve?

What if your problem is quality? How do you convince the business that Agile will result in a higher quality product? It’s not quite so easy. Agile itself won’t deliver better quality, but the good practices you’ll have to implement in order to successfully be agile will help to improve your quality. I was thinking about this the other day because it’s exactly the problem I was faced with.

Agile isn’t going to make it easier to reliably test our software. But to be agile, we need to be able to build and deploy our project rapidly so that we can test it right there and then, not tomorrow, not next week, but right now, so that the testers and devs can work in tandem, building features and signing them off and moving on to the next one. We have to facilitate this in order to be agile, so as a byproduct of going agile we might have to invest in creating a new build and deployment system. And it has to be quick so it’ll have to be automated.

So we have an automated build and deployment system, but to be able to reliably test our features we’ll have to make sure the environments are reliable. We can throw people at this problem and dedicate a team to making sure our environments are clean and regularly audited, or we can automate all that as well! chef Fortunately there are numerous tools and good practices we can follow to do this, just take a look at Chef, Puppet, Vagrant, and VMWare as examples of tools for automating deployments of virtual machines, and the concepts of “infrastructure as code” for good practices. (of course, if your hardware isn’t already virtualised the first thing to do is see whether it can be, and if it really, honestly can’t, then look at tools like Norton Ghost and Powershell for ways of automating as much as you can).

“Agile” and “Improved Quality” might not be the most obvious bed partners, but the journey to becoming agile almost forces you to take steps which will naturally go towards improving your quality.

Hopefully you’ll have enough “sales material” to put forward a great case for agile – you can deliver exactly what the customer wants, to a higher quality, and you can manage their expectations in a way you could never do before. And that’s just scratching the surface of what Agile can do for a business, but for the purposes of keeping this post to a reasonable length, I’ll leave it at those 3 things!

Phase 2 – Pick the most appropriate project, and start doing Scrum

The sales pitch is over and now it’s time to start doing stuff. Make your life a lot easier by picking a project that has as many of the following features as possible:

  • Smart developers and testers
  • Isn’t suffering from a tonne of technical debt
  • Has users who are happy to get involved in early & regular feedback
  • Is small, new or yet to begin

If you’re taking on an existing project, a good idea at this point is to benchmark your existing processes. Consider trying to measure the following:

  • How long does it take to get a single change from request through to production deployment?
  • How much time and money does it cost to fix an issue on production?
  • How many bugs do you typically find on your production code every month?
  • How often do you deliver features that don’t satisfy the customer?
  • How often do you deliver features after the deadline?

Measuring some of the things above is clearly non-trivial, but if you can find these stats somewhere, they’ll be very useful benchmarks for you in the future. When you can demonstrate that all of these metrics are improved in your new Agile process, god-like status will soon follow.

You - after delivering "agile" to the business

Guess who just delivered a release on time…
Ohhhh Yeeeeaaaaahhhh!

I recommend doing Scrum because it’s simple and has the most support in terms of people with experience, material (books, courses etc), and tools. It’s a good “framework” to get you started, and once you’ve had success, you can evolve into other methods, or incorporate them into Scrum (such as BDD, TDD etc).

Succeeding With Agile

Here’s one of many great books to get you started on your agile journey

At about this point you’ll need to do some brainwashing training. The concept of doing analysis, design, development and testing all at the same time is going to sound absolutely bonkers to some people. Try your best to explain it to them, but don’t waste too much time on this – just crack on and make a start!

Most people will enjoy the experience of working in this “new” way, and the first few sprints will probably benefit from the fact that everyone is performing better simply because they feel more invigorated. Use this opportunity to promote scrum across the organisation.

In this phase, always maintain a focus on “the business” and not just on the technical team. It’s important that the business feels part of this new process or they’ll just see it as some crazy dev thing which doesn’t really affect them, and they won’t try to understand it. Business people might refer to this as “Promoting Synergy”, which I’ve just shoe-horned into this post so that I can add a picture from Lonely Island’s “Like A Boss” video. However, I do like to make a point of always highlighting the extra business value we’re delivering, and make sure the Product Managers (soon to be “Product Owners”) are involved all the way. They represent the traditional link between the customers and what we’re delivering, and so it’s essential that they understand the benefits of agile.

Promote Synergy!

Promote Synergy!

I was recently asked about the impact of “going agile” on a project’s release schedule, and when we would be able to deliver the features we’ve promised to the customers. It’s difficult to explain that we no longer know when we’ll deliver stuff, but at some point, people will have to realise that this is the wrong question. I prefer the idea of a rolling roadmap, which is continually reviewed and updated (as often as you can afford to do it, really). Rolling Roadmaps give the business, as well as the customers a good idea of our intentions, but it is very different to fixed dates on a release schedule, or a traditional yearly roadmap. Of course, everyone needs to understand that the main driver for our deliverables will be the customers, and what the customer wants will usually change over a shorter period than you expect. So for your new “Agile” project, try to work towards implementing a rolling roadmap culture, and move away from long-term fixed delivery dates (if you can).

One final note on Phase 2: Make it fun, and make it different.

Phase 3 – Improve

Agile promotes “fast feedback loops” all over the place: in development we get fast feedback on our code through Continuous Integration, with BDD we get fast feedback to the Product Owners/BAs and of course with our more frequent releases we get faster feedback from the customers. And so it is with our Agile processes as a whole. With short sprints and the clever use of retrospectives we can continually tinker with our fine tuning to see if we can improve our quality and velocity. Look at areas you can try to improve, change something and then see if your change has had a positive impact at the end of the sprint. This is basically the concept behind Deming’s Shewhart Cycle:

demingcycleDeming actually preferred “Plan, Do, Study, Act”, whereas I myself prefer “Plan, Do, Measure, Act”. The reason I prefer this is because it implies the use of quantifiable metrics to base our actions on, rather than some other non-quantifiable observations. Anyways, the point is that after agile is applied, you should keep looking at ways to continuously improve. This is key to keeping everyone feeling fresh and invigorated, helps us to learn from our mistakes, and encourages innovation.

So there you go, Agile delivered in 3 well easy steps. It shouldn’t take you much longer than an afternoon. Ok, it might take a bit longer but if you’re looking for a 30,000 foot overview of a simple 3-phase approach, then you could do a lot worse than apply the principles of “Sell the Agile Idea, Pick the Best Project, and Keep Improving”.

Infrastructure Automation and the Cloud

As I write this, I’m sitting in a half-empty office in London. It’s half empty, you see, because it’s snowing outside, and when it snows in London, chaos ensues. Public transport grinds to a complete halt, buses just stop, and the drivers head for the nearest pub/cafe. The underground system, which you would think would be largely unaffected by snow, what with it being under ground, simply stops running. The overground train service has enough trouble running when it‘s sunny, let alone when it’s snowing. And of course most people know this, so whenever there’s a risk of snow, many people simply stay at home, hence the half-empty office I find myself in.

Snow in Berlin - where for some strange reason, the whole city doesn't grind to a standstill

Snow in Berlin – where for some strange reason, the whole city doesn’t grind to a standstill

But why does London grind to such a standstill? Many northern European cities, as well as American ones, experience far worse conditions and yet life still runs fairly normally. Well, one reason for London’s regular winter shutdown is the infrastructure (you can see where I’m going with this, right?). The infrastructure in London is old and creaking, and in desperate need of some improvement. The problem is, it’s very hard to improve the existing infrastructure without causing a large amount of disruption, thus causing a great deal of inconvenience for the people who need to use it. The same can often be said about improving IT infrastructure.

A Date With Opscode chef

Last night I went along to one of the excellent London Continuous Delivery Meetups (organised by Matthew Skelton at thetrainline.com – follow him on twitter here) which this month was all about Infrastructure Automation using Chef. Andy from Opscode gave us a demo of how to use Chef as part of a continuous delivery pipeline, which automatically provisioned an AWS vm to deploy to for testing. It all sounded fantastic, it’s exactly what many people are doing these days, it uses all the best tools, techniques and ideas from the world of continuous delivery, and of course, it didn’t work. There was a problem with the AWS web interface so we couldn’t actually see what was going on. In fact it looked like it wasn’t working at all. Anyway, aside from that slight misfortune, it was all very good indeed. The only problem is that it’s all a bit utopian. It would be great if we could all work on greenfield projects, or start rewriting everything from scratch, but in the real world, we often have legacy systems (and politics) which represent big blockers on the path to getting to utopia. I compare this to the situation with London’s Infrastructure – it’s about as “legacy” as you can possibly get, and the politics involved with upgrading it is obvious every time you pick up a newspaper.

In my line of work I’ve often come across the situation where new infrastructure was required – new build environments, new test server, new production environments and disaster recovery. In some cases this has been greenfield, but in most cases it came with the additional baggage of an existing legacy system. I generally propose one or more of the following:

  1. Build a new system alongside the old one, test it, and then swap it over.
  2. Take the old system out of commission for a period of time, upgrade it, and put it back online.
  3. Live with the old system, and just implement a new system for all projects going forward.

Then comes the politics. Sometimes there are reasons (budget, for instance) that prevents us from building out our own new system alongside the old one, so we’re forced into option 2 (by far the least favorable option because it causes the most amount of disruption).

The biggest challenge is almost always the Infrastructure Automation. Not from a technical perspective, but from a political point of view. It’s widely regarded as perfectly sensible to automate builds and deployments of applications, but for some reason, manually building, deploying and managing infrastructure is still widely tolerated! The first step away from this is to convince “management” that Infrastructure Automation is a necessity:

  • Explain that if you don’t allow devs to log on to the live server to change the app code, then why is it acceptable to allow ops to go onto servers and change settings?
  • Highlight the risk of human error when manually configuring servers
  • Do some timings – how long does it take to manually build your infrastructure – from provisioning to handover (including any wait times for approval etc)? Compare this to how quick an automated system would be.

Once you’ve managed to convince your business that Infrastructure Automation is not just sensible, but a must-have, then it’s time for the easy part – actually doing it. As Andy was able to demonstrate (eventually), it’s all pretty straightforward.

Recently I’ve been using the cloud offerings from Amazon as a sort of stop-gap – moving the legacy systems to AWS, upgrading the original infrastructure by implementing continuous delivery and automating the infrastructure, and then moving the system back onto the upgraded (now fully automated and virtualised) system. This solution seems to fit a lot more comfortably with management who feel they’ve already spent enough of their budget on hardware and environments, and are loath to see the existing system go to waste (no matter how useless it is). By temporarily moving to AWS, upgrading the old kit and processes, and then swapping back, we’re ticking most people’s boxes and keeping everyone happy.

Cloud Hosting vs Build-it-Yourself amazon_AWS

Cloud hosting solutions such as those offered by Amazon, Rackspace and Azure have certainly grown in popularity over the last few years, and in 2012 I saw more companies using AWS than I had ever seen before. What’s interesting for me is the way that people are using cloud hosting solutions: I am quite surprised to see so many companies totally outsourcing their test and production environments to the cloud, here’s why:

I’ve looked into the cost of creating “permanent” test labs in the cloud (with AWS and Rackspace) and the figures simply don’t add up for me. Building my own vm farm seems to make far more sense both practically and economically. Here are some figures:

3 Windows vms (2 webservers, 1 SQL server) minimum spec of dual core 4Gb RAM:

Amazon:

  • 2x Windows “Large” instance
  • 1x Windows “large” instance with SQL server
  • Total: £432 ($693.20)

Rackspace:

  • 3x 4Gb dual core = £455
  • 1x SQL Server = £o
  • Total: £455

Rackspace

These figures assume a full 730 hours of service a month. With some very smart time and vm management you could get the rackspace cost down to about £300 pcm. However, their current process means you would have to actually delete your vms, rather than just power them off, in order to “stop the clock” so to speak.

So basically we’re looking at £450 a month for this simple setup. Of course it’s a lot cheaper if you go for the very low spec vms, but these were the specs I needed at the time, even for a test environment.

The truth is, for such a small environment, I probably could have cobbled together a virtualised environment of my own using spare kit in the server room, which would have cost next to nothing.

So lets look at a (very) slightly larger scale environment. The cost for an environment consisting of 8 Windows vms (with 1 SQL server) is around £1250 per month. After a year you would have spent £15k on cloud hosting!

But I can build my own vm farm with capacity for at least 50 vms for under £10k, so why would I choose to go with Rackspace or Amazon? Well, there are actually a few scenarios where AWS and Rackspace have come in useful:

1. When I just wanted a test environment up and running in  no time at all – no need to deal with any ITOps team bottlenecks, just spin up a few vms and we’re away. In an ideal world, the infrastructure team should get a decent heads up when a new project is on it’s way, because the dev & QA team are going to need test environments setting up, and these things can sometimes take a while (more on that in a bit). But sadly, this isn’t an ideal world, and quite often the infrastructure team remain blissfully unaware of any hardware requirements until it’s blocking the whole project from moving forward. In this scenario, it has been convenient to spin up some vms on a hosted cloud and get the project unblocked, while we get on and build up the environments we should have been told about weeks ago (I’m not bitter, honestly :-))

2. Proof of concepting – Again no need to go through any red-tape, I can just get up and running on the cloud with minimal fuss.

3. When your test lab is down for maintenance/being rebuilt etc. If I could simply switch to a hosted cloud offering with minimal fuss, then I would have saved a LOT of downtime and emergencies in 2012. For example, at one company we hosted all our CI build servers on our own vm farm, and one day we lost the controller. We could have spun up another vm but for the fact that with one controller down, we were over capacity on the others. If I could have just spun up a copy of my Jenkins vm on AWS/Rackspace then I would have been back up and running in short order. Sadly, I didn’t have this option, and much panic ensued.

The Real Cost of Build-it-Yourself

So I’ve clearly been of the mind that hosting my own private cloud with a VMware VSphere setup is the most economically sensible solution. But is it really? What are the hidden costs?

Well last night, I was chatting with a couple of guys in the London Continuous Delivery community and they highlighted the following hidden costs of Build-it-Yourself (BIY):

Maintenance costs – With AWS they do the maintenance. Any hardware maintenance is done by them. In a BIY solution you have to spend the time and the money keeping the hardware ticking over.

Setup costs – Setting up a BIY solution can be costly. The upfront cost can be over £20,000 for a decent vm farm.

Management costs – The subsequent management costs can be very high for BIY systems. Who’s going to manage all those vms and all that hardware? You might (probably will) need to hire additional resources, that’s £40k gone!

So really, which solution is cheapest?

 

A Really $h!t Branching Policy

“As a topic of conversation, I find branching policies to be very interesting”, “Branching is great fun!”, “I wish we could do more branching” are just some of the comments on branching that you will never hear. And with good reason, because branching is boring. Merging is also boring. None of this stuff is fun. But for some strange reason, I still see the occasional branching policy which involves using the largest number of branches you can possibly justify, and of course the most random, highly complex merging process you can think of.

Here’s an example of a really $h!t branching policy:

Look how hateful it is!! I imagine this is the kind of conversation that leads to this sort of branching policy:

“Right, let’s just work on main and then take a release branch when we’re nearly ready to release”

“Waaaaait a second there… that sounds too easy. A better idea would be to have a branch for every environment, maybe one for each developer as well, and we should merge only at the most inconvenient time, and when we’ve merged to the production branch we should make a build and deploy it straight to Live, safe in the knowledge that the huge merge we just did went perfectly and couldn’t possibly have resulted in any integration issues”

“Er, what?”

“You see! Its complexity is beautiful”

Conclusion

Branching is boring. Merging is dull and risky. Don’t have more branches than you need. Work on main, take a branch at the latest possible time, release from there and merge daily. Don’t start conversations about branching with girls you’re trying to impress. Don’t talk about branches as if they have personalities, that’s just weird. Use a source control system that maintains branch history. Floss regularly. Stretch after exercise.

 

 

 

Why do we do Continuous Integration?

Continuous Integration is now very much a central process of most agile development efforts, but it hasn’t been around all that long. It may be widely regarded as a “development best practice” but some teams are still waiting to adopt C.I. Seriously, they are.

And it’s not just agile teams that can benefit from C.I. The principles behind good C.I. can apply to any development effort.

This article aims to explain where C.I. came from, why it has become so popular, and why you should adopt it on your development project, whether you’re agile or not.

Back in the Day…

Are you sitting comfortably? I want you to close your eyes, relax, and cast your mind back, waaay back, to 2003 or something like that…

You’re in an office somewhere, people are talking about The Matrix way too much, and there’s an alarming amount of corduroy on show… and developers are checking in code to their source control system….

Suddenly a developer swears violently as he checks out the latest code and finds it doesn’t compile. Someone’s check-in has broken the codebase.

He sets about fixing it and checking it back in.

Suddenly another developer swears violently….

Rinse and repeat.

CI started out as a way of minimising code integration headaches. The idea was, “if it’s painful, don’t put it off, do it more often”. It’s much better to do small and frequent code integrations rather than big ugly ones once in a while. Soon tools were invented to help us do these integrations more easily, and to check that our integrations weren’t breaking anything.

Tests!

Fossilized C.I. System

Fossil of a Primitive C.I. System

Excavations of fossilized C.I. systems from the early 21st Century suggest that these primitive C.I. systems basically just compiled code, and then, when unit tests became more popular, they started running unit tests as well. So every time someone checked in some code, the build would make sure that this integration would still result in a build which would compile, and pass the unit tests. Simple!

C.I. systems then started displaying test results and we started using them to run huuuuge overnight builds which would actually deploy our builds and run integration tests. The C.I. system was the automation centre, it ran all these tasks on a timer, and then provided the feedback – this was usually an email saying what had passed and broken. I think this was an important time in the evolution of C.I. because people started seeing C.I. as more of an information generator, and a communicator, rather than just a techie tool that ran some builds on a regular basis.

Information Generator

Management teams started to get information out of C.I. and so it became an “Enterprise Tool”.

Some processes and “best practices” were identified early on:

  • Builds should never be left in a broken state.
  • You should never check in on a broken build because it makes troubleshooting and fixing even harder.

With this new-found management buy-in, C.I. became a central tenet of modern development practices.

People started having fun with C.I. plugging lava lamps, traffic lights and talking rabbits into the system. These were fun, but they did something very important in the evolution of C.I. –  they turned it into an information radiator and a focal point of development efforts.

Automate Everything!

Automation was the big selling point for C.I. Tasks that would previously have been manual, error-prone and time-consuming could now be done automatically, or at night while we were in bed. For me it meant I didn’t have to come in to work on the weekends and do the builds! Whole suites of acceptance, integration and performance tests could automatically be executed on any given build, on a convenient schedule thanks to our C.I. system. This aspect, as much as any other, helped in the widespread adoption of C.I. because people could put a cost-saving value on it. C.I. could save companies money. I, on the other hand, lost out on my weekend overtime.

Code Quality

Static analysis and code coverage tools appeared all over the place, and were ideally suited to be plugged in to C.I. These days, most code coverage tools are designed specifically to be run via C.I. rather than manually. These tools provided a wealth of feedback to the developers and to the project team as a whole. Suddenly we were able to use our C.I. system to get a real feeling for our project’s quality. The unit test results combined with the static analysis could give us information about the code quality, the integration  and functional test results gave us verification of our design and ensured we were making the right stuff, and the nightly performance tests told us that what we were making was good enough for the real world. All of this information got presented to us, automatically, via our new best friend the Continuous Integration system.

Linking C.I. With Stories

When our C.I. system runs our acceptance tests, we’re actually testing to make sure that what we’ve intended to do, has in fact been done. I like the saying that our acceptance tests validate that we built the right thing, while our unit and functional tests verify that we built the thing right.

Linking the ATs to the stories is very important, because then we can start seeing, via the C.I. system, how many of the stories have been completed and pass their acceptance criteria. At this point, the C.I. system becomes a barometer of how complete our projects are.

So, it’s time for a brief recap of what our C.I. system is providing for us at this point:

1. It helps us identify our integration problems at the earliest opportunity

2. It runs our unit tests automatically, saving us time and verifying or code.

3. It runs static analysis, giving us a feel for the code quality and potential hotspots, so it’s an early warning system!

4. It’s an information radiator – it gives us all this information automatically

5. It runs our ATs, ensuring we’re building the right thing and it becomes a barometer of how complete our project is.

And we’re not done yet! We haven’t even started talking about deployments.

Deployments

Ok now we’ve started talking about deployments.

C.I. systems have long been used to deploy builds and execute tests. More recently, with the introduction of advanced C.I. tools such as Jenkins (Hudson), Bamboo and TeamCity, we can use the C.I. tool not only to deploy our builds but to manage deployments to multiple environments, including production. It’s now not uncommon to see a Jenkins build pipeline deploying products to all environments. Driving your production deployments via C.I. is the next logical step in the process, which we’re now calling “Continuous Delivery” (or Continuous Deployment if you’re actually deploying every single build which passes all the test stages etc).

Below is a diagram of the stages in a Continuous Delivery system I worked on recently. The build is automatically promoted to the next stage whenever it successfully completes the current stage, right up until the point where it’s available for deployment to production. As you can imagine, this process relies heavily on automation. The tests must be automated, the deployments automated, even the release email and it’s contents are automated.So what exactly is the cost saving with having a C.I. system?

Yeah, that’s a good question, well done me. Not sure I can give you a straight answer to that one though. Obviously one of the biggest factors is the time savings. As I mentioned earlier, back when I was a human C.I. machine I had to work weekends to sort out build issues and get working code ready for Monday morning. Also, C.I. sort of forces you to automate everything else, like the tests and the deployments, as well as the code analysis and all that good stuff. Again we’re talking about massive time savings.

But automating the hell out of everything doesn’t just save us time, it also eliminates human error. Consider the scope for human error in a system where some poor overworked person has to manually build every project, some other poor sap has to manually do all the testing and then someone else has to manually deploy this project to production and confidently say “Right, now that’s done, I’m sure it’ll work perfectly”. Of course, that never happened, because we were all making mistakes along the line, and they invariably came to light when the code was already live. How much time and money did we waste fixing live issues that we’d introduced by just not having the right processes and systems in place. And by systems, of course, I’m talking about Continuous Integration. I can’t put a value on it but I can tell you we wasted LOTS of money. We even had bugfix teams dedicated to fixing issues we’d introduced and not caught earlier (due in part to a lack of C.I.).

Conclusion

While for many companies C.I. is old news, there are still plenty of people yet to get on board. It can be hard for people to see how C.I. can really make that much of a difference, so hopefully this blog will help to highlight some of the benefits and explain how C.I. has been adopted as one of the most important and central tenets of modern software delivery.

For me, and for many others, Continuous Integration is a MUST.

 

Read-only Gradle Wrapper Files Are Bad, mmkaay

I’ve just been messing around with a Gradle build using Gradle Wrapper, trying to import an ant build which does some clever stuff. I couldn’t get it to work, so I eventually changed the Ant script to just echo a variable, and then just tried importing the ant file into gradle and calling the echo task, like this:

ant file (test.xml):

    <target name=”test” >
<echo>the value of main.version is ${main.version}</echo>
</target>

Gradle file:

ant.importBuild ‘test.xml’

And I was just running this:

gradlew test

Simples, right? Well, as it happens, no. I’m running the 1.0 “release” of gradle wrapper (which is actually versioned as 1.0-rc-3, strangely enough). Whenever I tried to run my highly complicated build (ahem), I got this lovely error:

Could not open task artifact state cache (D:\development\ReleaseEngineering\main\commonBuildStuff\.gradle\1.0-rc-3\taskArtifacts).
> java.io.FileNotFoundException: D:\development\ReleaseEngineering\main\commonBuildStuff\.gradle\1.0-rc-3\taskArtifacts\cache.properties.lock (Access is denied)

Access is denied! Gah! Of course it is! Wait, why is access denied? Well, basically that file is read-only because it’s in source control and I’m using Perforce. I removed the read-only flag and the build worked. Problem temporarily solved. It’s going to be interesting to see how this is going to work in the C.I. system…

 

PowerCLI: Reverting CI Agents to Snapshot

My friend Ed’s capacity to automate stuff is quite awesome. Yesterday he automated a way of making our Continuous Integration system alert us when one of the agents went offline. This is already automated in our CI system, but it just wasn’t automated enough for Ed’s liking, so he wrote a script. His script will send us an email whenever an agent goes offline. I haven’t recieved any emails so far, so either the agents are all fine, or the script isn’t working – there’s no way to tell, so I expect Ed will automate a way of telling us whether the automated script has run successfully.

Then today, in the true spirit of “DevOps”, he tells me he has automated a way of reverting our CI agents to a snapshot and plugged it in to the CI system, for good measure. The CI agents are all VMs deployed by VMware, so Ed has used the PowerCLI plugin to do the automation.

Basically the script just iterates over a list of VMs which are in a particular resource pool, and reverts them all to a snaphot. Here’s the script itself:

connect-viserver myserver.mycompany.com -User username -Password secret

$vmcsv = import-csv $args[0]

ForEach($line in $vmcsv){
Get-VM $line.name | Get-Snapshot -Name $line.snap | Set-VM -VM $line.name -confirm:$false
Get-VM $line.name | Start-VM
}

disconnect-viserver -confirm:$false

import-csv looks something like this:

name, snap
linuxSvr1, snap1
xpSvr1, snap1
xpSvrIE7, snap1
w7SvrIE8, snap1
w7SvrIE9, snap1

Ed has added the execution of this script to our CI system, so any of the devs can revert their CI servers to a snapshot by simply pressing a button. They key thing here is to organise VMs into resource pools. We’ve got dedicated resource pools per dev team/project, so it’s safe enough to allow the devs to do this without running the risk of affecting anyone elses CI builds.

You can follow Ed on twitter (@ElMundio87) and check out his blog here: http://www.elmundio.net/blog/

Upcoming Agile/DevOps/CI Events

There’s a free talk this evening at Skills Matter (London) about Continuous Delivery by Tom Duckering and Marc Hofer. Tom did a talk on “Coping with Big CI” a few months ago, which was interesting and very well delivered. I’m looking forward to tonight’s talk.

Then tomorrow there’s the DevOps summit (again in London), which is being chaired by Stephen Nelson-Smith, author of “Test-Driven Infrastructure with Chef” (you can find my review of the book here). Atlassian and CollabNet will have speakers/panelists at this event so I’m expecting it to be very worthwhile.

On the 26th June, again in London (it’s all happening in London for a change), there’s Software Experts Summit, subtitled “Mastering Uncertainty in the Software Industry: Risks, Rewards, and Reality”, I’m expecting there will be a decent amount of DevOps/Continuous Delivery coverage. Speakers include representatives from Microsoft and Groupon.

Next Thursday (June 28th) there’s an Agile Evangelists Meetup in London entitled “Agility within a Client Driven Environment” with talks from experienced agile experts from a range of industries. These are usually pretty good events and the speakers usually have a great deal to offer.

And as I mentioned in an earlier post, there’s the Jenkins User Conference in Israel on July 5th.

Sonar Analysis Using Gradle

I’ve been experimenting with Gradle recently, and as part of the experiment, I wanted to get Sonar running and producing code metrics, including test coverage reports. I’m running the first release version of Gradle, so version 1.0.

To get Sonar working in Gradle you need to apply the sonar plugin, like this:

apply plugin: ‘sonar’

Then you need to add some sonar connection settings (very much like with Maven):

sonar {
server {
url = “http://${sonarBaseName}/”
}
database {
url = “jdbc:mysql://${hostBaseName}:3306/sonar?useUnicode=true&characterEncoding=utf8”
driverClassName = “com.mysql.jdbc.Driver”
username = “wibble”
password = “wobble”
}
}

To run the Sonar analysis/reports, you just call sonarAnalyze, which is the in-built task that the Sonar plugin gives you. So far, so easy.

The first problem was with the version of Sonar. My colleage Ed (check out his blog here) was trying to get a gradle build working with an existing Sonar installation, but wasn’t having much joy. We were using a version of Sonar pre version 2.8, so we had to upgrade. In the end we were forced to upgrade to version 3.0.1. That was the first pain point.

The next problem we stumbled upon was with cobertura. There’s a cobertura plugin for Gradle, and getting it to work is a bit unusual. You need to reference an initialisation script which is hosted on GitHub, like this:

buildscript {
apply from: ‘https://github.com/valkolovos/gradle_cobertura/raw/master/repo/gradle_cobertura/gradle_cobertura/1.2/coberturainit.gradle&#8217;
}

We had some problems with this. One day, I could access this script fine, and the next it failed. A week or so later, I could access it, but Ed’s build couldn’t. We still don’t understand why this was the case, but we suspect it was something to do with the GitHub https connection.

To make sure we didn’t get this problem again, we got hold of the initialisation script and saved it locally – unfortunately it has dependencies so we had to download the whole folder and put this in our artifactory repository, and make the build reference it from there. This seemed to fix our problem, but it left us with another issue – we were now depending on another build component, which contained hard coded build configuration information (the initialisation script refers to the maven central repo). We weren’t happy with this (since we use our own cached repositories in artifactory), so we had to think of a solution.

Ed went away to meditate on our problem. A little while later he came back with a gradle build file which used the Cobertura ant task. It’s pretty much the same way as it’s documented in the gradle cookbook, here.

These are the important parts that you need to include:

def cobSerFile="${project.buildDir}/cobertura.ser"
def srcOriginal="${sourceSets.main.classesDir}"
def srcCopy="${srcOriginal}-copy"
dependencies {
        testRuntime 'net.sourceforge.cobertura:cobertura:1.9.3'
        testCompile 'junit:junit:4.5'
}
test.doFirst  {
    ant {
        // delete data file for cobertura, otherwise coverage would be added
        delete(file:cobSerFile, failonerror:false)
        // delete copy of original classes
        delete(dir: srcCopy, failonerror:false)
        // import cobertura task, so it is available in the script
        taskdef(resource:'tasks.properties', classpath: configurations.testRuntime.asPath)
        // create copy (backup) of original class files
        copy(todir: srcCopy) {
            fileset(dir: srcOriginal)
        }
        // instrument the relevant classes in-place
        'cobertura-instrument'(datafile:cobSerFile) {
            fileset(dir: srcOriginal,
                   includes:"my/classes/**/*.class",
                   excludes:"**/*Test.class")
        }
    }
}
test {
    // pass information on cobertura datafile to your testing framework
    // see information below this code snippet
}
test.doLast {
    if (new File(srcCopy).exists()) {
        // replace instrumented classes with backup copy again
        ant {
            delete(file: srcOriginal)
            move(file: srcCopy,
                     tofile: srcOriginal)
        }
        // create cobertura reports
        ant.'cobertura-report'(destdir:"${project.buildDir.path}/reports/coverage",
format:'xml', srcdir:"src/main/java", datafile:cobSerFile)
ant.'cobertura-report'(destdir:"${project.buildDir.path}/reports/coverage",
format:'html', srcdir:"src/main/java", datafile:cobSerFile)
    }
}

So this is how we’ve got it running at the moment. As you can see, we’re no longer using the Cobertura plugin for gradle. The next thing we need to do is get Sonar to pick up the Cobertura reports. This is configured in the Sonar configuration section. I’ve shown the Sonar configuration section at the top of this page, but now we need to make some changes to it, like this:

sonar{

project {
coberturaReportPath = new File(buildDir, “/reports/cobertura/coverage.xml”)
sourceEncoding = “UTF-8”
dynamicAnalysis = “reuseReports”
testReportPath = new File(buildDir, “/test-results”)
}

server {
url = “http://${sonarBaseName}/”
}
database {
url = “jdbc:mysql://${hostBaseName}:3306/sonar?useUnicode=true&characterEncoding=utf8”
driverClassName = “com.mysql.jdbc.Driver”
username = “wibble”
password = “wobble”
}
}

Now we need to go back and change the output directory of our Cobertura ant configuration, to make it output to /reports/cobertura/coverage.xml, so we change the last bit of our configuration to look like this:

 // create cobertura reports

        ant.'cobertura-report'(destdir:"${project.buildDir.path}/reports/cobertura/coverage",
format:'xml', srcdir:"src/main/java", datafile:cobSerFile)
ant.'cobertura-report'(destdir:"${project.buildDir.path}/reports/coverage",
format:'html', srcdir:"src/main/java", datafile:cobSerFile)