8 Principles of Continuous Delivery

continuous deliveryDave Farley co-authored “Continuous Delivery”, an excellent book in the Martin Fowler signature series, which goes into great detail about the evolution of Continuous Integration, and how to achieve continuous delivery (or continuous deployment) using “build pipelines”.

I went along to hear Dave Farley give a talk on Continuous Delivery and how they’re doing it where he works, at LMAX. It was a really great session and he managed to cover, in quite a short session, a great deal of content from in the book. I’ve put together a highlight of what he covered in the talk, mixed with my own take on things

Here’s what I learned…

Continuous delivery is basically the logical extension of Continuous Integration  – it’s a more holistic solution than C.I. though, as it encapsulates a lot more than just the development of software.

For instance, continuous delivery focuses a lot more on requirements than C.I. ever did, and involves a great deal more people on the delivery chain than traditional C.I. as well. It also has a greater customer focus than C.I.

Now, here’s something I didn’t know about continuous delivery…

There are 12 principles behind the agile manifesto. the first of which is:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software

Well who’d have thought it? Continuous delivery was mentioned waaaaay back in the days of the agile manifesto, some 2500 years BC* and yet for most of us it seems like a pretty new idea.

Continuous delivery is based on the use of smart automation. This is all about creating a repeatable and reliable process for delivering software. You have to automate pretty much everything in order to be able to achieve continuous delivery. manual steps will get in the way or become a bottleneck. This goes for everything from requirements authoring to deploying to production.

The focus is on the finished article – again, this is described as being:

Working software in the hands of the user

software in the hands of the user

software in the hands of the user

Because the focus is on the software in the hands of the user, there’s less tendency from a developers perspective, to simply chuck software over the wall to the QA team, and similarly to the Netops/production team.

Continuous delivery is all about getting that product out there, and getting the feedback from the users. This might mean delivering “unfinished” demo software during your development iterations, and getting your users to give valuable early feedback, or it might mean deploying experimental software to a website cluster and tracking how successful this new site is as compared to the existing system. Either way, it’s all about feedback loops. Essentially you want to have as rapid a feedback loop with your users as possible.

Feedback loops are familiar to everyone who has worked on a Continuous Integration system. In C.I. feedback loops are generally about getting test feedback (unit test, acceptance test, performance test etc) as quickly as possible – “Fail Fast” – as you’ve probably heard.

Continuous Delivery, as described, takes this idea to it’s logical conclusion, and gets the users involved in the feedback loop. This is a good example of how Continuous Delivery is more holistic than its C.I. predecessor. In Continuous Delivery, the feedback loop provides feedback not only on the quality of your code, but on the quality of your requirements, and the quality of your processes for delivering software.

8 Principles of Continuous Delivery

  1. The process for releasing/deploying software MUST be repeatable and reliable. This leads onto the 2nd principle…
  2. Automate everything! A manual deployment can never be described as repeatable and reliable (not if I’m doing it anyway!). You have to invest seriously in automating all the tasks you do repeatedly, and this tends to lead to reliability.
  3. If somethings difficult or painful, do it more often. On the surface this sounds silly, but basically what this means is that doing something painful, more often, will lead you to improve it, probably automate it, and this will eventually make it painless and easy. Take for example doing a deployment of a database schema. If this is tricky, you tend to not do it very often, you put it off, maybe you’ll do 1 a month. Really what you should do is improve the process of doing the schema deployments, get good at it, and do it more often, like 1 a day if needed. If you’re doing something every day, you’re going to be a lot better at it than if you only do it once a month.
  4. Keep everything in source control – this sounds like a bit of a weird one in this day and age, I mean seriously, who doesn’t keep everything in source control? Apparently quite a few people. Who knew?
  5. Done means “released”. This implies ownership of a project right up until it’s in the hands of the user, and working properly. There’s none of this “I’ve checked in my code so it’s done as far as I’m concerned”. I have been fortunate enough to work with some software teams who eagerly made sure their code changes were working right up to the point when their changes were in production, and then monitored the live system to make sure their changes were successful. On the other hand I’ve worked with teams who though their responsibility ended when they checked their code in to the VCS.
  6. Build quality in! Take the time to invest in your quality metrics. A project with good, targeted quality metrics (we could be talking about unit test coverage, code styling, rules violations, complexity measurements – or preferably, all of the above) will invariably be better than one without, and easier to maintain in the long run.
  7. Everybody has responsibility for the release process. A program running on a developers laptop isn’t going to make any money for the company. Similarly, a project with no plan for deployment will never get released, and again make no money. Companies make money by getting their products released to customers, therefore, this process should be in the interest of everybody. Developers should develop projects with a mind for how to deploy them. Project managers should plan projects with attention to deployment. Testers should test for deployment defects with as much care and attention as they do for code defects (and this should be automated and built into the deployment task itself).
  8. Improve continuously. Don’t sit back and wait for your system to become out of date or impossible to maintain. Continuous improvement means your system will always be evolving and therefore easier to change when needs be.

To go with these principles there are also:

4 Practices of Continuous Delivery

  1. Build binaries only once. You’d be staggered by the number of times I’ve seen people recompile code between one environment and the next. Binaries should be compiled once and once only. The binary should then be stored someplace which is accessible only to your deployment mechanism, and your deployment mechanism should deploy this same binary to each successive environment…
  2. Use precisely the same mechanism to deploy to every environment. It sounds obvious, but I’ve genuinely seen cases where deployments to QA were automated, only for the production deployments to be manual. I’ve also seen cases where deployments to QA and production were both automated, but in 2 entirely different languages. This is obviously the work of mad people.
  3. Smoke test your deployment. Don’t leave it to chance that your deployment was a roaring success, write a smoke test and include that in the deployment process. I also like to include a simple diagnostics test, all it does it check that everything is where it’s meant to be – it compares a file list of what you expect to see in your deployment against what actually ends up on the server. I call it a diagnostics test because it’s a good first port of call when there’s a problem.
  4. If anything fails, stop the line! Throw it away and start the process again, don’t patch, don’t hack. If a problem arises, no matter where, discard the deployment (i.e. rollback), fix the issue properly, check it in to source control and repeat the deployment process. A lot of people comment that this is impossible, especially if you’ve got a tiny outage window to deploy things to your live system, or if you do your production changes are done in the middle of the night while nobody else is around to fix the issue properly. I would say that these arguments rather miss the point. Firstly, if you have only a tiny outage window, hacking your live system should be the last thing you want to do, because this will invalidate all your other environments unless you similarly hack all of them as well. Secondly, the next time you do a deployment, you may reintroduce the original issue. Thirdly, if you’re doing your deployments in the middle of the night with nobody around to fix issues, then you’re not paying enough attention to the 7th principle of Continuous Delivery – Everybody has responsibility for the release process. Unless you can’t avoid it, I wouldn’t recommend doing releases when there’s the least amount of support available, it simply goes against common sense.

* Approximate date.

Ten Mistakes That Software Team Leads Make

“Ten Mistakes” (as I shall now call it because I’m too lazy to keep typing the whole title), was a talk by Roy Osherove which I went to at Skills Matter. He basically takes us through ten common mistakes he sees team leaders make, and offers some solutions to them. He also looks like Adam Sandler, I kid you not.

Roy Osherove

Roy Osherove

Adam Sandler

Adam Sandler

Here’s Roy delivering a piece to camera…

And here’s Adam, doing some funny acting, or something.

Adam Roy started us off by talking about a number of questions that team leaders might have. These included such things as:

  • How do I convince my team to do X
  • What do I do with the bad apple in my team?
  • What am I supposed to do as a team lead?
  • Why can’t we get away from fighting fires?
  • Will I lose friends?

He said that these were all questions that haunted him for years, and in the rest of the talk he goes on to explain how to answer them. He is also in the process of writing a book called “Notes to a Software Team Leader” which also covers these points.

So, we then moved on to the first of the Ten Mistakes…

#1 Not Recognising Team Maturity

This was a good place to start because much of the talk referred back to the “maturity level” of the team. Roy says that there are 3 levels of maturity when it comes to describing agile teams. These are:

  1. Chaos
  2. Learning
  3. Self-Leading

Chaos

A chaos team is one where people are always too busy. Maybe they’re always firefighting or they’re just being asked to too much in too little time, eithesinking shipr way, the result is the same – chaos. Nobody has any time to get organised and nobody has any time to learn anything new because they’re just too busy for that. This is obviously not a great maturity level to be at if you ask me, because eventually people will suffer burn out or get frustrated at the lack of opportunities to learn, and eventually the good guys will leave. However, Roy says that the chaos level is actually exceedingly common, and I can easily believe him. The trick is to act in an appropriate way if you’re the leader of a chaos team. A chaos team leader needs to be assertive and strong in their actions.

When the ship is sinking, you need a leader to give orders, not call a meeting

A chaos team leader will often need to take a stand and maybe tell management that the team cannot do everything that they’re being asked to do. It’s a tough role, it requires you to make tough decisions with conviction.

Management done right is a really tough job

So why, as a team leader, do you have to make all of these tough decisions yourself, instead of discussing them with your team? Well the simple answer is that there just isn’t enough time for meetings. by making these executive decisions yourself, you’re giving your team some breathing space, or simply just the space they need to get their work done. Sure, you might make a few wrong decisions, life’s tough, but it’s for the greater good because you’re giving your team the space they need to grow into the next level, a Learning Team.

Learning

This level of maturity is one in which the team has a greater degree of self organisation, but the team members still need to be coached. A team leader will need to grow his/her team by constantly challenging and questioning them, maybe even setting them homework! The goal with these teams is to improve week-on-week, and to get the team members to start solving their own problems.

So what are you going to do about it?

As a team leader of a Learning team you need to start to get your team members to solve their own problems in order to grow into a self-leading team. So if someone comes to you with a problem, encourage them to think of ways to fix their issue, and empower them to do so by responding with “so what are you going to do about it?”.

Self Leading

The third level of maturity is the self-leading team. This is where we all want to be! In this team, the leader is more like a mentor – he doesn’t tell people what to do or make executive decisions on behalf of the team as he/she would in a chaos team. Even in a self leading team, the team leader should still spend in excess of 50% of his time with his team.

So, the first mistake in our list is to not recognise what maturity level your team is at, and therefore not know how to lead your team in the right way. If you run your team as if they were self-leading, but in reality they’re chaos, then before long you’ll be heading up a certain creek without a paddle.

#2 Fear of Delegation

If you’re used to taking things on and doing them yourself, then it can be quite hard to feel comfortable delegating work to people, especially if you have trouble relying on others to get the work done.

If everyone feels comfortable in what they’re doing, then you’re doing something wrong

When you delegate work, you’ll need to get used to asking other people to take responsibility for something you’d otherwise do yourself. This responsibility can take some people out of their comfort zone, and this is a good thing. It’s important to challenge your team and get them out of their comfort zone as it’ll help them grow.

#3 Fear of Engagement

This generally means not communicating effectively, but Roy breaks this down even further:

#4 Placating

“The Bus Factor” – what’s that? Well, the bus factor is the number of people needed to get hit by a bus for the project to grind to a standstill. It’s all about individuals holding lots of information. I’ve seen this in a number of places, on good projects as well as bad, so I think it’s fairly natural, but the point that Roy makes is that you mustn’t placate these individuals just because they hold a lot of crucial knowledge. A person with a bus factor of 1 (meaning they could bring the project to a halt if they got hit by a bus) should be treated just the same as anyone else. I love the idea of people having a bus factor, just because it reminds me of the Kevin Bacon number.

#5 Being Irrelevant

I think this is about being in too many meetings and dealing with too many emails etc and so on – basically not being there for the team, losing contact with the real work, and generally being irrelevant. Nothing to do with Kevin Bacon.

#6 Being Super-reasonable

Not sure I really agree with the terminology here but Roy says that it’s super-reasonable to assume that everyone understands what your talking about when in reality you might not be getting your point across. I think the point to make here is that when you’re dealing with groups of people, as in an agile team, it’s wrong to assume they all have the same level of knowledge and understanding as yourself, and that you should communicate with them in the best way suitable, which tends to be by not making too many assumptions.

#7 Blaming

If you make your mind up that someone’s rubbish, then you will consciously and subconsciously use this as an excuse to not engage that individual. There will always be some people who are rubbish, but what you need to do is work on their weaknesses and bring them up to the level of the team, rather than to avoid engaging them, as this just means you’ll constantly be carrying a dead weight.

#8 Ignoring Behaviour Forces

You must understand the forces that act on a person in order to understand how they behave. There are 3 main types of forces that act on a person:

  • Personal
  • Social
  • Environmental

All of these can affect a teams ability to be successful, and you need to work out exactly what these forces are, and determine whether they are affecting your teams ability. An example of an environmental force is not having enough hardware to do what you need to do – for instance, if you don’t have budget for a continuous integration server then it’s going to be almost impossible to be agile.

#9 Fear of Assertiveness

Apparently this is common in the UK and in Norway, but not in Denmark. Bet you didn’t know that. Well it’s true, allegedly. Anyway, assertiveness is all about standing your ground and not living with things that you feel aren’t acceptable. If your team is in chaos mode, then you need to be especially assertive. A fear of being assertive can be disastrous in a chaos team.

#10 Spreading Non-Commitment

This is about using vague language. Roy says that you should always commit to deadlines, and when speaking to your team, make sure they tell you when they will do something by. Apparently, just by making the commitment in the words they use, they will be more motivated to deliver on that commitment. Roy suggests that when you have a meeting, end it by asking people what they’re actions are, and make sure they answer in the form “I will do X by Y”. However, people should only commit to doing things that are under their control, there’s no point committing to doing something that you have to get someone else to do. Also, as soon as you know that you’re unable to deliver on time, let the team know and they might be able to help you out and maybe then you will be able to deliver on time.

Questions and Answers

The Q&A went on for an age. I’ll summarise in bullet form, because bullet points are great!

  • You need to recognise when to change your leadership style – you need to stop being a chaos team leader if the team moves on the being a learning team
  • There’s no such thing as a chaotic learning team. The 2 cannot co-exist, however, teams can switch from one form to another.
  • Overseas teams don’t work as well as having everyone in-house. If you’re in this situation, you need to “change your reality”.
  • Agile teams should be 2 pizza teams – i.e. only as large as can be fed by 2 pizzas.
  • Good teams are grown, not hired
  • Scrum sometimes doesn’t fit teams in chaos mode
  • There’s no difference between a team leader and a manager if they’re the same person, which they can be.
  • Protect your team from project managers if your in the chaos mode

Books

A number of books were mentioned. Here they are!

Managing Teams Congruently – Gerald M Weinberg

Managing Teams Congruently

Behind Closed Doors – Johanna Rothman

Behind Closed Doors

Influencer (The Power To Change Anything) – Patterson et al

Influencer - The Power To Change Anything

Succeeding With Agile – Mike Cohn

Succeeding With Agile

Changing File Permissions On Files in Perforce

I had a script file which I had written on my Windows machine, which I wanted to execute on the build server. However, the build server couldn’t execute it because it didn’t have execute permissions by default, and I couldn’t change this on my windows machine.

So, what I had to do was check out the file on my linux box, using a clientspec I’d created, and edit the permissions like this:

p4 -c my_clientspec_test edit -t text+x jamestest.sh

my_clientspec_test is the name of the clientspec I used

jamestest.sh is my script file, obviously.

-t  tells the system to open the file as a particular type – in this case we pass text+x, which means text, with the “executable” modifier.

Of course, if your file isn’t text type you need to change the command to suit, i.e. -t binary+x if it’s a binary.