Digitalisation. Four: Agile Delivery

Digitalisation in a Post-Covid19 World (4/6)

Previously in digitalisation...

Welcome to the fourth in the series on Digitalisation and the Digital Maturity Model aimed at helping demystify an often-confusing realm for leaders, by translating technical terms from the argot of engineer-speak to logical constructs that help leaders navigate modern computing concepts.

The first article provided a backdrop as to why digitalisation will accelerate post-Covid19, set out the six aspects of the Digital Maturity Model, and covered the first of these, Cloud Platforms. The second article covered Cloud Native Applications, whilst the third discussed DevOps and Site Reliability Engineering (SRE). In this, the fourth in the series, we turn the lens to Agile Delivery, a much misunderstood concept.

Defining chaos

It’s long been recognised that for a programme of change to be successful, a methodology is required. Methodologies provide a framework for delivery that includes a way to organise the work, a way to allocate it, manage risk, manage costs, cater for testing, provide for acceptance, and govern delivery. No small ask.

A problem with methodologies is sheer volume of them. One of the problems with choice is that, like so much in life, people tend to want to stick with what they know and will seek ways of justifying that stickiness, a.k.a. confirmation bias. This leads to much debate (some of it quite heated) as to which methodology is best. But is this helpful? Is there another way? As it turns out, no it’s not that helpful because yes there is another way. It involves the classification of problems and using the right tool for the job.

Problems Problems Problems

At its heart, Change is about solving problems. In the second in this series, I touched upon the concept of Wicked Problems and provided a list of attributes that can be used to define them. But there are other classes of problem – another two to be precise: Tame and Critical. Let’s summarise each type:

1: Tame Problems

A problem you’ve seen before – Déjà Vu – is a tame problem. It may be complex with multiple moving parts, but the solution is known. An example is that of an infrastructure project. You know all of the parts that are needed, when they need to slot together, who needs to do what, and so on.

2: Critical Problems

There’s no time to sit down and think – fais le maintenant – The problem has to be solved now (incredibly near-term). Covid19 provides a great example of this. As countries went into lockdown many firms had to scrabble around to acquire kit and implement tools and policies to enable staff to work from home. There wasn’t time for planning in many cases.

3: Wicked Problems

You’ve never seen this problem before – Vu jàdé. There’s no clear understanding of either the problem or the solution, there’s no defined stopping rule, you may need a collective view to understand how to interpret it, and there may be many interdepencies between the solution to this problem and others. This is the main class of problems in software engineering, regardless of the platform.

Grouping Further

So, we have three ways to classify problems. Wouldn’t it be good if we could find three methodologies for solving them? It turns out that this is exactly what we have – the plethora of methodologies mentioned earlier fall into three neat categories: JFDI (or JDI if you prefer your acronyms to be three letters rather than four), Waterfall, and Agile. Let’s have a quick look at the history of each.

Myth Understanding

JFDI

Let’s start with JFDI (Just F@#%ing Do It). This is a command and control methodology which is very simple to summarise: We don’t have time to do anything other than what I tell you, so just do it. Hmmm… sounds a bit like what’s needed for Critical Problems? Indeed.

Waterfall

Okay, what about Waterfall? It was first documented in 1970 by Winston Royce, who criticised it in a number of ways, most notably by stating that:

Before continuing with:

Royce didn’t actually name Waterfall. That appears to have been left to a 1976 paper (see below), but the images in his paper were certainly christened that by those who read it. His words seem, at first sight, quite damming. Not to mention, we’ve all been there: cost overruns, delivery not matching requirements, etc. When it happens, it’s not pretty. The problem is that there are so many moving parts, external dependencies, and understandable lack of clarity over what the sponsor really wants at the beginning of so many projects that such overruns are to be expected when using Waterfall. But, there is a class of project to which Waterfall is superbly well suited. When you know the desired outcomes and the desired dependencies, Waterfall turns out to be a great methodology. Hang on… What did I say about problems? Oh yes, Tame problems fit that bill quite nicely. So, got a Tame problem? Use Waterfall.

Agile

Rather neatly, this leave us with one class of problem (Wicked) and one methodology (Agile). Indeed, with the whole Fail Fast approach that Agile encourages, it is an excellent methodology for solving Wicked Problems. Recall that earlier I said that software engineering problems, in the main, tend to be Wicked Problems. This is because it’s very difficult to pin down exactly what you want at the beginning (incidentally why Waterfall projects looking to solve Wicked Problems are so often mired in large numbers of Change Requests). The stopping condition is usually not known (which is why WF projects tends to come in releases / phases / drops / etc.). Product evolution occurs over the product’s lifecycle. This is because neither client expectations nor external systems stand still.

the Change Methodology Selection Flowchart

Let’s pull this into a handy tool to help you choose which methodology is the right one for your project. The Change Methodology Selection Flowchart:

How to Get Agile Wrong

Great. So you’ve selected Agile, now how do you ensure you get it right, especially when your team are well versed in Waterfall or have only tangentially encountered Agile. Let’s start with some common mistakes:

Just give people a Kanban board, a tool like Jira. If they use that they’ll be doing Agile.
Take a Project Manager and call them, instead, a SCRUM Master.

These are the second and third most frequent mistakes firms make. Can you guess the first? Worry not, I shan’t keep you guessing, it is (ta dah!): Lack of training.

Lack of training is a great mistake because it doesn’t actually require that much upfront investment to get team members to a proficient standard to enable them to make valuable contributions. It can be done in as little as a week per person when done properly. That’s not to say that your team will be experts at the end of that time, but they’ll be making informed and valuable contributions.

The problem is that when people aren’t trained, they make mistakes which are often costly. These include, using multiple Kanaban boards to represent items in a project (duplication of effort); thinking that Kanban board columns are fixed, thereby constraining workflow capabilities; assuming that a SCRUM master is just another name for a Project Manager; believing that Sprints are not important units of work, often instead referring to sprints as subcomponents of releases; and forgetting that change is not about an orderly list of activities, but, importantly, also requires a large amount of the C word (all is revealed in the sixth of five episodes)

Here are a couple more mistakes:

Believing that Scaled Agile (of which a number of versions exist) means creating a standardised Agile model that must be inflexibly imposed on the organisation. In effect, all this is is Waterfall rebadged, often by large consultancies whose incentive model is based on selling firms large numbers of resource. We’ll discuss poor incentives elsewhere in a later article.
Managers / primary stakeholders not attending demos / show and tell meetings. This is common but it means that managers and stakeholders don’t get to see the progress. This, in turn, leads to a lack of confidence in the process, especially if, several iterations down the line, the stakeholder wants to change something that was delivered a while back; a lack that need not exist.

We're risk averse - Agile's Too New

Many believe that Agile is fairly new. After all, the Agile Manifesto only popped into existence back in 2001. This is true, but what Agile is is a collective noun for a host of different methodologies that can be traced right back to the 1940s, when Spitfires were built using the Two-Bin system. Components were placed into two bins for workers to assemble. When one became empty that was the signal to refill it, thus ensuing no interruption to the flow of work. It was then taken up by Toyota in the late 1940s and eventually grew to become the system we know today. Many other aspects of Agile were well established by 2001. For example, Small Self-Managed Teams. Richard Pascale, in his book Managing On the Edge: How Successful Companies Use Conflict to Stay Ahead, originally published in 1990 and revised many times since, has documented many of these (you can read more here, including a limited update since 2001). So, we can see that, in fact, not only is Agile not new, it arguably predates Waterfall.

Getting Agile Right

Before you go down the path of sending everyone on a course, I would advise getting experts in to help evaluate what agile means in your context. For some people Sprints, SCRUMS, and daily stand-ups will be the order of the day. For others, continuous integration, continuous deployment will fit the bill (some firms deploy hundreds of times per day – small incremental changes that can be easily undone). Getting experts in to discuss is a very good first step.

Once you’ve chosen your flavour of Agile, then think about training and using experts to complement your team until the experts can be rolled off. It will pay dividends and save a lot of money in lost time and delays.

Going forward

The truth is that Agile can be a great tool for value creation if used properly, but when firms make the mistake of doing it the wrong way – as outlined above – it adds no value. If you want complex software engineering problems (Wicked Problems) to be solved in a way that lowers your spend, brings flexibility, and complements the rapid rate of change that using Public Cloud brings, Agile is where you need to be. Whether you bring in experts or not, get your troops trained and make sure you include the leadership team so they understand what their teams are talking about.

Additional Resources

Web Resources

As a reaction to the model that saw firms held to hostage by giant consultancies that had little interest in providing value in client-led timeframes, a number of firms have sprung up that offer outcomes based consulting models to help drive the market towards providing better value for clients (better value for spend, as one client of StoryPositive’s so eloquently puts it). Of course, we would love to engage with you on the leadership piece – helping leaders to navigate the complexity and set the goals and outcomes for those they engage – but we’re not the engineers, the pre-formed teams of specialists. We have worked with the following companies to provide successful outcomes on Agile-based Delivery. Here is a selection:

Adaptavis – A company focussed on Agile methods, efficiency, and the removal of Delay Costs
eSynergy Solutions – Delivering pre-formed software engineering teams and projects related to DevOps Transformation, Cloud Computing and Data Engineering.
Think Above Cloud – A firm focussed on delivering high-value change through Public Cloud engineering
Secure Delivery – Experts in Application Security Consulting, Training, and Assessment.

Books and publications

The following books are worth investigating:

Our Iceberg Is Melting, John Kotter and Holger Rathgeber
That’s Not How We Do It Here, by John Kotter and Holger Rathgeber
Thinking Fast and Slow, Daniel Kahneman
Managing On the Edge: How Successful Companies Use Conflict to Stay Ahead, by Richard Pascale, 1990 (and updated many times since)
Dilemmas in Design Thinking, by Horst Rittel and Melvin Webber, 1973.
Wicked Problems and Clumsy Solutions: the Role of Leadership, by Professor Keith Grint, 2008
Managing the development of large software systems, by Dr Winston W. Royce, 1970.
Software Requirements: Are they Really A Problem, T. E. Bell and T. A. Thayer, 1976.

Note, there are no links to the books. I’ll leave it to you to source a copy however you prefer rather than try and earn a few pennies from a commission-based link. Articles are linked where possible.

Before you Go...

If you’ve found this article informative, please consider sharing it with colleagues and others in your network.

Next time...

In the next article, we’ll cover the fifth aspect of Digital Maturity: The Operating Model. Don’t worry, it’s also aimed at leaders, not engineers.