The DevOps Team Myth

August 13, 2014 1 comment

Should “DevOps” be a job title? Are DevOps teams an anti-pattern? Can you do DevOps within a single team? Were the moon landings staged in the Arizona desert? These are the sort of questions you’re never more than 5 feet away from at any DevOps conference or meetup. The answers, of course, are:

  • Yes
  • Yes and no
  • Maybe
  • Don’t be silly

Everyone seems to have an opinion on the whole “DevOps Team/Job Title” question, and I’m no different, except my opinion is this:

I passionately don’t care

Let me explain:

If you have a devops team, and their job is to encourage and develop a devops culture within the organisation, then that’s perfectly fine, surely? If that team is successful and a devops culture blossoms, then you’re definitely winning. Maybe that team could subsequently change its name, but that just seems a bit pointless and overly hung-up on semantics. In this case, I don’t care what the team is called, because it doesn’t matter.

If you have a devops team, and they don’t encourage or foster a devops culture within the organisation, then you’re doing devops wrong, as simple as that. If you’re doing devops wrong then it doesn’t matter what you call a team, you’re still doing it wrong. In this case, I don’t care what the team is called, because it doesn’t matter. 

I do understand the opinions of the anti-pattern crowd. Setting up a separate silo responsible for “doing devops” is completely wrong. It discourages other teams and individuals from adopting a devops attitude, as they’ll see it as someone else’s responsibility. But the problem isn’t with the existence of the team, the problem is the purpose of the team and the fact that whoever decided the team should “do DevOps” clearly doesn’t understand what DevOps is.

At the very core of the issue is an understanding of what DevOps actually is. If your CTO, (or Head of Technology or whoever calls the shots within the technology division) thinks that DevOps just means automating builds and deployments, treating Infrastructure as Code, or adopting Continuous Delivery, etc then you’ve already got a problem. DevOps is about these things (and more), sure, but it’s about everyone understanding the importance of them, and absorbing them into their culture.

I’ve previously posted about the dangers of having an obsession with Ownership and Responsibility and I think these factors can also contribute to a failed adoption of devops. Drawing up clear lines of ownership and responsibility is risky – if you get it wrong, you’re going to struggle. For instance, if you draw a clear line of ownership around “devops” and place it firmly in the domain of the devops team, then you’re not going to do devops. A single team cannot own and be responsible for a culture, it just doesn’t work like that. The lines of ownership need to be blurred or wiped out entirely. Nobody and everybody should be “responsible for” and “own” the devops culture.

Conclusions On The DevOps Team

Anyone who thinks that you can get something ingrained into an organisation’s culture by setting up an isolated group of people who are solely responsible for doing those things, is flying in the face of conventional wisdom. Even my experience suggests that this doesn’t work (and I always fly in the face of conventional wisdom, coz I’m stooopid). By all means create a DevOps Team, but don’t make them responsible for “doing devops”, that’s just wrong. Instead, give them the challenge of spreading the DevOps gospel, evangelising DevOps within the organisation and training everyone on the benefits of devops, as well as some of the tricks of the trade.

On The DevOps Job Title

Everyone’s a devops engineer these days. I’m a devops engineer, my wife’s a devops engineer, even my dog’s a devops engineer. It’s a booming industry and everyone wants a piece of it. I’m afraid we don’t have any control over this anymore, we’ve created a beast and it’s consuming everything!! Luckily it’s not the end of the world. Sure, sysadmins the world over are now rebranding themselves as devops engineers, but does it make any difference at the end of the day? If you’re hiring, you don’t just hire someone on the strength of their previous job title do you? No, you actually read their CV and interview the candidate. Good candidates will always shine through. Working out if someone’s a sysadmin or a build engineer is just a bit more hassle for recruitment agents, that’s all!

My dog, the DevOps Engineer

My dog, the DevOps Engineer

In a way I’m thankful for the devops job title. I honestly think it has helped to make the whole devops thing more popular and opened it up to a wider audience.

Should there really be such a thing as a “DevOps Engineer”? Probably not, but we’re far too late to stop it, and trying to stop it seems a bit of a waste of energy to me. Eventually “DevOps Engineer” will come to mean something more specific, but for now we’re just going to have to read a few more lines on CVs.

 

 

Stop Comparing Software Delivery With Manufacturing!

July 10, 2014 1 comment

A couple of weeks ago I was at an Experience Devops event in London and I was talking about how software delivery, which is quite often compared to a manufacturing process, is actually more comparable to a professional sports team. I didn’t really get time to expand on this topic, so I thought I’d write something up about it here. It all started when I ran a cheap-and-nasty version of Deming’s Red Bead Experiment, using some coloured balls and an improvised scoop…

The Red Bead Experiment

I was first introduced to Deming’s Red Bead Experiment by a guy called Ben Mitchell (you can find his blog here). It’s good fun and helps to highlight how workers are basically constrained by the systems the work in. I’ll try to explain how the experiment works:

  • You have a box full of coloured beads
  • Some of the beads are red
  • You have a paddle with special indentations, which the beads collect in (or you could just use a scoop, like I did).
  • You devise a system whereby your “players” must try to collect exactly, let’s say, 10 red beads in each scoop.
  • You record the results

Now, given the number of red beads available, it’s unlikely the players will be able to collect exactly 10 beads in each scoop. In my especially tailored system I told the players to keep their eyes closed while they scooped up the balls. I also had about half as many red beads as any other colour (I was actually using balls rather than beads but that doesn’t matter!). The results from the first round showed that the players were unable to hit their targets. So here’s what I did:

  • Explain the rules again, very clearly. Write them down if necessary. Being as patronising as possible at this point!
  • Encourage the players individually
  • Encourage them as a team
  • Offer incentives if they can get the right number of red beads (free lunch, etc)
  • Record the results

Again, the results will be pretty much the same. So…

  • Threaten the individuals with sanctions if they perform badly
  • Pick out the “weakest performing” individual
  • Ask them to leave the game
  • Tell the others that the same will happen to them if they don’t start hitting the numbers.

In the end, we’ll hopefully realise that incentivising and threatening the players has absolutely zero impact on the results, and that the numbers we’re getting are entirely a result of the flawed system I had devised. Quite often, it’s the relationship between workers and management that gets the attention in this experiment (the encouragement, the threats, the singling out of individuals), but I prefer to focus on the effect of the constraining system. Basically, how the results are all down to the system, not the individual.

Thanks Kanban!

I think one of the reasons why the software industry is quite obsessed with traditional manufacturing systems is because of the Toyota effect. I’m a huge fan of the Toyota Production System (TPS), Just-in-time production (JIT) Lean manufacturing and Kanban – they’re all great ideas and their success in the manufacturing world is well documented. Another thing they all have in common is that various versions of these principles have been adopted into the software development world. I also happen to think that their application in the software development world has been a really good thing. However, the side-effect of all this cross-over has been that people have subconsciously started to equate software delivery processes with manufacturing processes. Just look at some of the terminology we use everyday:

  • Software engineering 
  • Software factories
  • Kanban
  • Lean
  • Quality Control (a term taken directly from assembly lines)

It’s easy to see how, with all these manufacturing terms around us, the lines can become blurred in people’s minds. Now, the problem I have with this is that software delivery is NOT the same as manufacturing, and applying a manufacturing mindset can be counter-productive when it comes to the ideal culture for software development. The crucial difference is the people and their skillsets. Professionals involved in software delivery are what’s termed as “knowledge workers”. This means that their knowledge is their key resource, it’s what sets them apart from the rest. You could say it’s their key skill. Manufacturing processes are designed around people with a very different skillset, often ones that involve doing largely repetitive tasks, or following a particular routine. These systems tend not to encourage innovation or “thinking outside of the box” – this sort of thing is usually assigned to management, or other people who tend not to be on the production line itself. Software delivery professionals, whether it be a UX person, a developer, QA, infrastructure engineer or whatever, are all directly involved in the so-called “production line”, but crucially, they are also expected to think outside of the box and innovate as part of their jobs. This is where the disconnect lies, in my opinion. The manufacturing/production line model does NOT work for people who are employed to think differently and to innovate.

If Not Manufacturing Then…

Ok, so if software delivery isn’t like manufacturing, then what is it like? There must be some analogous model we can endlessly compare against and draw parallels with, right? Well, maybe…

 

home sweet home

home sweet home

I’m from a very rural area of west Wales and when anyone local asks me what I do, I can’t start diving into the complexities of Agile or devops, because frankly it’s so very foreign to your average dairy farmer in Ceredigion. Instead, I try to compare it with something I know they’ll be familiar with, and if there’s one thing that all people in west Wales are familiar with, it’s sheep rugby.

It’s not as daft as it sounds, and I’ve started to believe there’s actually a very strong connection between professional team sports and Agile software development. Here’s why:

Software delivery is a team effort but also contains subject matter experts who need to be given the freedom to put their skills and knowledge to good use – they need to be able to improvise and innovate. Exactly the same can be said of a professional rugby or soccer (yes, I’m going to call it soccer) teams. Rugby and soccer are both team sports but both contain very specific roles within that team, and for the teams to be successful, they need to give their players the freedom and space to use their skills (or “showing off” as some people like to call it).

2008 World Player of the Year Shane Williams

2008 World Player of the Year Shane Williams

Now, within a rugby team you might have some exceptionally talented players – perhaps a winger like former World player of the year Shane Williams. But if you operate a system which restricts the amount of involvement he gets in a game, he’ll be rendered useless, and the team may very well fail. Even with my dislike of soccer, I still think I know enough about how restrictive formations and systems can be. The long ball game, for instance, might not benefit a Lionel Messi style player who thrives on a possession & passing game.

The same can be said of software delivery. If we try to impose a system that restricts our individual’s creativity and innovation, then we’re really not going to get the best out of those individuals or the team.

 

So Where Does Agile Fit Into All of This?

Agile is definitely the antidote to traditional software development models like Waterfall, but it’s not immune from the same side-effects as we witness when we do the red bead experiment. It seems to be that the more prescriptive a system is, the greater the risk is of that system being restrictive. Agile itself isn’t prescriptive, but Kanban, XP, Scrum etc, to varying degrees are (Scrum more prescriptive than Kanban for instance). The problem arises when teams adopt a system without understanding why the rules of that system are in place.

prescriptive = restrictive

For starters, if we don’t understand why some of the rules of Scrum (for instance) exist, then we have no business trying to impose them on the team. We must examine each rule on merit, understand why it exists, and adapt it as necessary to enable our team and individuals to thrive. This is why a top-down approach to adopting agile is quite often doomed to fail.

So What Should We Do?

My advice is to make sure everyone understands the “why” behind all of the rules that exist within your chosen system. Experiment with adapting those rules slightly, and see what impact that change has on your team and on your results. Hmmm, that sounds familiar…

The Deming Cycle: Plan, Do, Check, Act

The Deming Cycle: Plan, Do, Check, Act

 

Categories: Agile Tags: , , , , ,

On collective ownership and responsibilities

July 3, 2014 2 comments

Recently I’ve been butting heads with some people on the subject of Ownership, Responsibility and Accountability.  There seems to be a very unhealthy obsession with these things sometimes, and I think this is indicative of a less-than-ideal culture. I don’t want to say that they’re “anti-agile” because that just sounds a bit weak, and because I also think they’re not just bad for agile, they’re bad for pretty much any system. I’m not sure how familiar most people are with the “RACI matrix” concept, but in my eyes it’s downright evil in the wrong hands, and I’ve been hearing “RACI Matrix” a lot recently (it’s now on my Bullshit Bingo card).

 

I’ll start off by clarifying what I mean. I’ve got nothing against people owning actions or being accountable for certain particular (usually small) things, but I do take offence when pretty much everything has to be given an owner, someone accountable and someone to “take responsibility”. It’s divisive and results in lots of finger pointing, in my experience.

 

I much prefer the concept of shared ownership, and collective accountability. As a software delivery team, we should all feel responsible for the quality of the product, as well as the performance and the feature richness. These things shouldn’t be assigned for ownership to individuals, as it’ll create an attitude of “well it’s not my problem” among the other team members.

Here’s an example: I’ve worked in a team where one person was made the “owner” of the build system. They busied themselves making sure all the builds passed and that the system was regularly ticking over. Of course, the builds often failed and nobody cared except this one person, who then had to try to get people to fix their broken builds. It almost seemed as if people didn’t care about the fact that their software wasn’t capable of being compiled, or that the tests were failing, and in truth they didn’t. They cared about writing code and checking it in, because they didn’t “own” the build system.

 

One message that I always try to drive home with software delivery teams is that our objective is to make software that works for our users, not just write code. I know how easy it is for developers to just focus on checking in code, or perhaps just make sure it passes the tests in the CI system, but beyond that, their focus drops off. I know because I was once one of those developers :-) These days I try to encourage everyone to care about things such as:

  • How your code builds
  • How the tests execute
  • How good the tests are
  • How good the code is
  • How easy it is to deploy
  • How easy it is to maintain
  • How easy it is to monitor

Because it takes all of these things to produce good software that users can enjoy, which means we get paid.

 

Here’s another example of how “ownership” has hurt a product: A large system I once worked on was deployed into production using a complicated system of bash and perl scripts, which were cobbled together by a sysadmin who did the deployments. He became the de facto “owner” of the deployment system. There were untold issues with the running of the application because of permissions, paths etc and so forth. The deployment process was creaky and relatively untested. Since the “ownership” of this system was assigned to the sysadmin, rather than devolved or collectively shared throughout the delivery team, the “deployability” was seen as a second class citizen within the delivery team, because everybody felt like it was “owned” by one person who just happened to be on the periphery of the team at best.

 

So here’s what I think: The ability to monitor, maintain, deploy, test, build and create software should all be treated as first class citizens and should be the collective responsibility of everyone in the team. They should all own it, and they should all be accountable.

I would extend this out further, to include supporting systems such as environments, build systems, testing frameworks and so-on. Sure, each team might have an SME or two who focuses more on one of these things than any other, but that doesn’t make that one person accountable, responsible or the owner any more than any particular developer is the “owner” of any particular class, method or function. If I write some code that depends on a method that someone else has written, and that method is failing, I don’t just down tools, shrug my shoulders and say “well I’m not accountable for that”. That would be hugely unhelpful and I’d make no friends either. In the same way, we shouldn’t treat our supporting functions and systems as someone else’s responsibility. If we need it in order to make our software work for the end user, then it’s our collective responsibility, no matter what “it” is.

Categories: Agile Tags: ,

How to move a VM image from one storage account to another in Azure

June 23, 2014 2 comments

Well, this was a painful experience. I googled until my fingers were sore, and even when I thought I got the right solution, it didn’t quite work for me. Anyway, here’s what I wanted to do:

I had a storage account in West Europe, but some bright spark decided to create our virtual network in North Europe, so I had to move one of my disk images (a 127GB Windows 2008 image) from West to North. 

The first thing I needed to do was create a new Storage Account (I called it DiskImages) in the correct target location, namely North Europe.

The next thing I did was make the container in my source Storage Account public, otherwise the command I was going to run would fail. I made this change via the UI (go to your source storage account, then select the relevant container and click edit). I didn’t have to do this for the target Storage Account though, and I’m way too weary to work out why (probably because you end up passing the Access Key in the command later).

Oh I nearly forgot, I needed to install and configure the Azure Cross-Platform CLI (you can find details here), because having only one command line interface (Azure Powershell) with your Azure subscription just isn’t enough!

The last thing I needed was to copy my Access Key for my target Storage Account (just go to the storage account and click on “Manage Access Keys” at the bottom).

Then I ran this command:

azure vm disk upload https://SOURCE_STORAGE_ACCOUNT_URL.blob.core.windows.net/vhds/win2k8-win2k8-2014-05-15.vhd https://DiskImages.blob.core.windows.net/vhds/win2k8-win2k8-2014-05-15.vhd gv5hQZGJuOPFJWsSuFFiCiEnTLYgooFFEdArouNDWITH4nptTg==

And it worked!

So, basically that’s just “azure vm disk upload [SOURCE] [TARGET] [TARGET_ACCESS_KEY]“

That’s when I realised that I was copying a 127GB image from 1 datacentre to another and that:

a) It would take about 4 hours

b) It would cost money

And that’s when I stopped it, and just made a new template image in the correct location. You live and learn.

Categories: DevOps Tags: ,

Connecting your azure environment to your office VPN

June 11, 2014 3 comments

Okay, before I go anywhere with this topic I should point out that:

a) This is most definitely NOT a step-by-step guide on how to configure your VPN device

b) This is basically just an overview of stuff you need to know before you start

c) I can’t think of a third thing to put here, but 2 things just doesn’t feel like enough to justify a list

Why on earth would you need to connect your azure environment to your office VPN anyway?

Actually there’s all sorts of reasons for doing this, for instance you might need your Azure hosted services to connect directly to servers/services inside your office VPN. My main reason for needing to do this was to connect my Azure VMs to my Chef server running on a VM inside the office VPN. (“Why not just move your Chef server to Azure as well?!” I hear you ask. Well, let’s just imagine there was a really good reason for this, and move on).

Setting up a VPN connection can be a bit of a pain (and take ages to implement) with some datacentre providers, but with Azure it’s actually rather quite easy. The first thing you need to determine is the type of VPN connection you want to set up. Your 2 main options are point-to-site and site-to-site.

Point-to-Site essentially just involves setting up a virtual network within Azure and connecting out to it from individually configured clients within your office (if you ever work from home and VPN into the office network then you’ll be very familiar with this type of setup).

Site-to-Site involves connecting an existing office VPN to a virtual network within Azure (it’s basically the equivalent of adding your Azure subscription to your local office network).

I opted for a site-to-site connection because it scales well, and once it’s set up there’s no need to use VPN clients on my on-premise servers.

If you want to setup a site-to-site VPN connection to Azure you’ve basically got 2 choices:

  • Setup a connection between your existing VPN hardware (you can find a list of supported VPN devices here) and an Azure Virtual Network
  • Setup a connection between an Azure Virtual Network and a local Windows 2012 R2 server with Routing and Remote Access Service (RRAS).

Setting up a connection using your existing VPN hardware

Many organisations will have dedicated VPN devices, but as mentioned previously not all of these are suitable for connecting a site-to-site VPN to Azure. If your device does happen to be supported then you’ll need to get hands-on with the device configuration in order to setup the site-to-site connection. This will differ from one device to the next, so good luck with that!

Whatever supported device you’re using, you’ll still need to create and configure a virtual network in Azure. The full instructions on how to do this can be found here, but here’s a basic checklist of the sort of stuff you’ll need to know:

  • Your DNS Servers
  • Your local network name (obvs)
  • Your VPN device’s IP address
  • Your address space
  • Subnet details (if you want to create one)
  • Affinity group name (you can create one as you go through the Virtual Network setup)

Other than creating the virtual network, you just need to create a gateway within that virtual network. Details of how to do that can be found here. This stuff is all really simple from within the Azure Management UI.

And that’s about it from the Azure side. You now just need to configure your office VPN device. As mentioned earlier, the details of how to do this will depend on what device you have, so time to dig out your VPN device’s user manual!

 

But what if your VPN device isn’t on “The List”??

Well, fear not, for there is another way! All you need is a Windows 2012 Server with RRAS configured.

NOTE: I know you can also configure RRAS on Windows server 2008 R2 but I don’t yet know if this will work (we’re still trying to test it out as I’m writing this). Here, try this guide if you fancy giving it a shot, and let me know if it works with Azure!

One thing to note is that the Microsoft documentation pretty much says this setup won’t work if your RRAS server is behind a NAT or a firewall, but this isn’t actually the case. It’ll work just as long as your RRAS server has a public IP address.

So, here’s a basic overview of what you’ll need:

  • The same shizzle as previously for the Azure Virtual Network
  • A Windows server 2012 with 2 NICS
  • A public IP address on the 2012 server
  • A local Gateway server (you could just use the RRAS machine for this though)
  • ICMPv4  enabled on your firewall

So there we are, nothing too complicated at all. There’s plenty of configuration work to be done in setting all this stuff up, but the Azure side is definitely the easy part. As for the RRAS stuff, don’t install and configure this manually – you actually need to edit a powershell script with the details you get along the way, and then run the script. It sounds like a ball-ache, but it’s actually more fun than the usual Windows service installation! There are plenty of good resources for helping you work through a site-to-site setup in a step-by-step guide, such as:

 

 

 

 

 

 

 

Categories: DevOps Tags: , , , ,

Enabling winrm using powershell

May 12, 2014 1 comment

So, you’re doing stuff with these new “virtual” machines eh? Well if you’re using windows, there’s a damn good chance you’ll need to enable and configure winrm, otherwise you won’t be able to log in to your swanky new “virtual machine”! Even Chef needs this service running on the target in order to work with windows. Anyway, here’s what to do: open a powershell prompt and type the following:

winrm quickconfig -q

winrm set winrm/config/winrs ‘@{MaxMemoryPerShellMB=”512″}’

winrm set winrm/config ‘@{MaxTimeoutms=”1800000″}’

winrm set winrm/config/service ‘@{AllowUnencrypted=”true”}’

winrm set winrm/config/service/auth ‘@{Basic=”true”}’

Start-Service WinRM

set-service WinRM -StartupType Automatic

Alternatively you could create a ps1 script containing the stuff above, open powershell, do the thingy that allows you to run unsigned scripts, namely:

Set-ExecutionPolicy Unrestricted

Then run the ps1 script.

There, I’ve blogged it, now I’ll never have to google this again!

Are “Ready For” Columns on Kanban Boards The Enemy of God?

April 17, 2014 5 comments

This is going to be a quick rant post, hopefully. Today I saw another Kanban board which had a “Read for test” column on it, here’s the screenshot:

Not 1 but 2 "Ready for" columns!

Not 1 but 2 “Ready for” columns!

 

I Think “Ready For” Columns Are Baaaaad

With most Kanban boards you mark a card as done when it’s ready to be pulled into another column. If that means it has to be deployed before a card is ready for test then so be it. The last thing we want is cards just sitting around waiting – this is baaaaaad. “Ready for Test” usually means it’s either deployed (and yet to be tested) or waiting to be deployed. Either way, not much is happening to the work sitting in this column. Basically it’s waste (or “muda” as the Lean Kanban aficionados might call it), and remember, waste is baaaaad.

 

Seriously, I Think They’re Baaaaad

A result of using these “Ready For x” columns is that they tend to slightly move us away from the “stop the line” practice that good Lean/Kanban systems employ. Basically whenever there’s a problem, or a bottleneck is appearing, we want to stop the production line and address the issue. So, if we keep all these “Ready for QA” cards in our In Dev or Code Review Column (basically whatever column comes before your Ready for QA column) then we’ll very quickly reach our WIP (Work In Progress) limit and the line will be stopped. That’s a good thing! We want to catch that bottleneck as soon as we can, we don’t want to hide it by pushing our cards into another “buffer” column.

 

Did I Mention That I Think “Ready For” Columns in Kanban Are Baaaaaad?

Yet another problem with “Ready for x” columns is that they quite often tend to be push rather than pull columns. You can’t really pull into a Ready for QA column as it isn’t an actual “workflow” state, it’s a “wasteflow” state (see what I did there?). I mean, who’s going to pull stuff into that column anyway? I’ve yet to meet a “ready for test” team who just sit around pulling cards into their column before marking them as “ready” (presumably once they’ve verified that they are indeed ready for test). Ok, you might have a deployment team who are responsible for deploying stuff to your test environments and so forth. In this case, your workflow state still isn’t “Ready for test” it’s “In Deployment”.

 

Conclusion

“Ready for x” columns make baby Jesus cry.

Categories: Agile Tags: , ,
Follow

Get every new post delivered to your Inbox.

Join 418 other followers