DevOpsNet

I was at DevOps World last week (nothing like Disney World, by the way) and happened to be paying attention to a talk by a chap called Jonathan who worked at Barclays Bank. He briefly mentioned a couple of KPIs that they measure to track the success of their DevOps initiative. He mentioned these:

• Lead Times
• Quality
• Happiness
• Outcomes

This list looked quite good to me, I thought “They sound pretty sensible, I’ll remember those for the next time someone asks me about DevOps KPIs”. The reason I thought this, you see, is because I get asked “What are good DevOps KPIs?” almost every week. Colleagues, clients, friends & family, random strangers, the dog… Everyone asks me. It’s like I’m wearing a T-Shirt that says “Ask me about DevOps KPIs” or something.

So, the time has come to formulate a decent answer. Or, more specifically, write a blog on it, so I can then tell people to read my blog! Hurrah!

A couple of months ago, while discussing a DevOps transformation with a global telecomms company, the subject of metrics and KPIs came up. We’d spent the previous hour or so hearing about how one particular part of the business was so unique and different to all the others, and that any DevOps transformation would need to be specifically tailored to accommodate this business’s unique demands. I totally agree with this approach. However, when the subject of KPIs came up, the “one-size-fits-all” approach was favoured.

It’s common for organisations to want KPIs that span the whole organisation. It’s convenient and allows management to compare and contrast (for whatever good that’ll bring). But does this “one-size-fits-all” approach work? Or does it encourage the wrong behaviours?

You can’t manage what you can’t measure

Personally, I think you need to be very careful about selecting your KPIs and metrics. Peter Drucker once observed that “you can’t manage what you can’t measure”, which sounds sensible enough, but this leads us towards trying to measure everything (because we want to manage as much as we can, right?). But that’s where things get a bit tricky. As soon as we start measuring things, they change – this is known as Goodhart’s Law. But what I’m talking about specifically is people changing their behaviours because they’re being measured.

Once you measure something, it changes

If we’re being measured on utilisation level, we try to expand our work to fill the time we have available, in order to look fully utilised. It’s what people do! By doing this, people lose the “downtime” they used to have, the time when people are most creative, and as a result, innovation suffers.

So what should we measure?

It depends on what you’re trying to achieve, and what side-effects you’re able to tolerate. Think very carefully about how your metrics and KPIs could be interpreted by both subordinates and management.

For example, I’m currently working with a team who until recently measured the age of stories in the backlog. The thought was, the larger the number, the longer it’s taking to get stuff done. The reality was different. In reality, there was an increasing number of low priority stories, which were often (and quite legitimately) overlooked in favour of higher priority stories. So what did the metric really prove? That the team were slow or that the team were effective at prioritising?

I think generally speaking that most stats need to be accompanied by a narrative, otherwise they’re open for misinterpretation. But we know that there’s often very little room for narrative, and that the fear of misinterpretation drives people to try to “game” the stats (that is to say, legitimately manipulate the results). And this is another reason why we have to be very careful when we’re planning KPIs and reporting metrics.

Data Driven Metrics

In 2014 Gartner produced a report entitled “Data Driven DevOps: Use Metrics to Help Guide Your Journey” in which they listed a range of typical DevOps metrics, categorised by type, such as “Business Performance”, “Operational Efficiency” and so on. I’ve picked out a few of the metrics in the table below. I’ve also added some others which I’ve been using in one form or another. This is by no means an exhaustive list of DevOps KPIs, but it might be somewhere to start if you’re looking for inspiration.

Measuring tangibles and intangibles

One thing to be conscious of is that you can’t really measure things like “culture” and “collaboration” directly. Culture, for example, is an intangible asset, and you can only really measure the result of Culture, rather than the culture itself. The same goes for collaboration.

In the table above, be conscious of things like “happiness”, “value” and “sharing” as these can sometimes be hard to measure directly, not to mention being somewhat subjective.

Working remotely is so common these days, that I’d say the vast majority of organisations I work with today accommodate some degree of remote working. I think it’s great that organisations are prepared to do this for the sake of their employees –  it shows an awareness of that so-called “work-life balance” that so many of us have managed to get wrong in the past.

It’s perhaps not surprising then, that when I speak to people about the importance of culture and collaboration as essential ingredients in any devops or agile transformation, people become concerned about how compatible this is with their working-from-home policy and globally distributed teams.

Unfortunately I’d say there’s no straight answer. As with most things in DevOps, it depends on many factors. We all love a good list, so here’s my list of things that can impact your DevOps journey if you’re working with distributed teams and remote workers:

• Team maturity (is there strong trust within the team?)
• Decision making (how effective are you at decision making?)
• Location (do your working days overlap much?)
• Language (do you all speak and understand the same language effectively?)
• Collaboration (It’s not the same as communication)
• Management techniques (are you a command and control freak?)
• Tooling (are you on mute?)

Bearing these factors in mind, let’s take a look at what makes an effective distributed team (in my humble opinion, of course).

High Trust

High trust between individuals in the team means people are comfortable allowing others to work autonomously, safe in the knowledge that they will deliver what they committed to. High trust also means you’ll feel comfortable asking for help when you need it. In high-trust teams, a daily stand-up is usually sufficiant for a Team Lead (Scrum Master, PM, PO, or whoever) to feel comfortable and confident that the individuals can be left to get their work done. This is not to say that they will work alone, far from it, to do DevOps successfully you absolutely MUST collaborate and work together effectively throughout the day, but crucially they don’t need a Team Lead to continually check in on them to make sure they’re doing the right things.

Devolved Decision Making

Responsibility, accountability, empowerment – all of these are high-scoring bullshit bingo words, but they’re also important factors in effective decision making. Give team members the power to make decisions without having to summon a committee meeting and things will run much more smoothly. If this idea scares you, then mitigate the perceived risk by ensuring all decisions are retrospectively reviewed – doing this allows people to go ahead and get on with their work but it also allows you to catch any bad decisions before it’s too late. When it comes to decision making within the team, my policy is to seek forgiveness, not permission. Of course, there’s a boundary within which we all need to work when it comes to decision making, and decisions around features etc should always be made by the Product Owner, but we all knew that, right? Obviously I’m not suggesting that individuals should unilaterally decide to change the language the product is coded in, or introduce a new feature – common sense still has a large role to play!

Location

Most of the successful distributed teams I’ve been involved with have had a significant amount of overlap in terms of the hours worked by the individuals. The least successful teams have had very little overlap in working hours. If you work in the UK and you have significant parts of your team in places such as the US West Coast, Australia, China or Japan then you’ve probably already felt the frustration of having to wait an entire working day for an answer or response from a colleague, only to find that it wasn’t what you needed. In the fast-paced IT world of today, many of us can ill afford that sort of delay, so teams have worked out new ways of dealing with the challenge, such as working different shifts, dialling in to meetings at unsociable hours and so on. It’s often not an ideal solution but if it helps the team work more effectively while allowing you to continue to enjoy a comfortable and convenient working lifestyle then it’s probably worth the effort.

In a DevOps environment you’ll want to make sure that there’s as much overlap as possible between your developers and infrastructure engineers – these are the roles that need to collaborate most closely, so an environment where ALL your devs are in one location and ALL of your infrastructure team are the other side of the world is going to be a real challenge.

Language

Ok, I’ll be blunt – you all need to speak the same language, and you need to do it well. We all know it’s hard work trying to communicate with people who don’t fluently speak the same language as you, and in the end you just end up making excuses for not communicating with them (and that’s a bad thing). It’s high-time we all agreed to speak 1 global business language, and that language should be Welsh (because it’s by far the most awesome language in the world).

Collaboration

For me collaboration is about people working together in an effort to build something mutually beneficial. It’s not the same as communication. Collaboration means you need to be able to listen to other people, make appropriate changes, help others, coach people and share ideas. Tools like GitHub (along with the Git workflows) are great for allowing us to collaborate when working on code. Teams with good collaboration techniques and processes (code reviews, retrospectives, workshops etc) tend to become higher trust teams as well. I haven’t stopped to think why this happens, but it’s an observation. These teams handle distributed working easily because they have such a high degree of interaction anyway, that location becomes insignificant. In a successful DevOps environment both developers and infrastructure engineers will collaborate and use techniques such as pairing and code reviews to learn from each other and improve.

Management Techniques

Again we’re looking at high-trust teams. Teams where management are happy to give the individuals the space and time to work are more effective than teams with managers who feel the need to constantly check in on them. In my experience the best style of management to work with a distributed team is one of high-trust and devolved responsibility – one where management provide guidance and support rather than instructions. If you see yourself as a command-and-control style manager or obsessed with micro-managing individuals then you’re probably going to struggle working with a distributed team.

Tooling

There’s loads of tooling out there to help people work remotely. Most people are already using things like Slack, HipChat and Skype because they are such effective communication tools – but communication is only part of the picture. As I mentioned earlier, GitHub is a great collaboration tool for anyone involved in coding (so devs and ops alike), but we also often need to share large binaries (such as PDFs, Presentations, diagrams, pictures  and so on) which don’t usually belong in source control alongside your code. For these types of artifacts tooling like Google Drive and Dropbox are great (as long as your corporate security policy will allow you to use them). I like the latest Atlassian tools for managing requirements and handling wikis because the real-time updates work really well with people working remotely, but in terms of sheer simplicity and ease of use, you can’t look any further than Trello for task management! I’ve seen IdeaBoardz being used very effectively for brainstorming and sharing ideas across a distributed team – like Trello it’s a really easy-to-use and fun collaboration tool.

So, in summary, doing DevOps in a distributed team can be an absolute doddle or it can leave you dead in the water – it all depends on how mature your team is, what sort of management you have, the tooling available to you, the communication skills of the individuals, and your team culture.