Which metrics assist in DevOps proving its ROI? Part One.

Having completed a (98 page) MSc dissertation on DevOps and Return On Investment (ROI), the intention was always to share my work with the IT community for common benefit.

However one clear message I heard during the research, was that noble (but lengthy) academic tomes were not the way to communicate with the busy IT people! So here we have a series of four “bite-sized” articles for your consumption.

Part One: Setting the Scene – and the challenges of measuring IT Success.

Despite some of the later articles in this series having a clear DevOps focus, there are messages that are applicable to all areas of IT investment and subsequent success measurement. So do persevere, as the outcomes may be helpful in your particular field!

Research Scope

The dissertation topic I have chosen looks at the challenges of measuring and proving return on IT investment. The investment I chose to examine is the transition to a DevOps application development to deployment methodology.

Being new, DevOps is under the spotlight to prove its ROI – and there are plaudits both for and against, which can make the debate appear quite polarized. Can the wisdom of academia assist with this debate and help with proving DevOps ROI?

The scope of the research is as follows:

Being so new, and with no one single definition, there is no one universally authored document which prescribes the characteristics of “good DevOps”. There is however a sizeable body of opinion from DevOps commentators (books, blogs, whitepapers etc) which provide good material.

  • When considering DevOps Key Success Factors (KSF) and pointers, what do the DevOps commentators prescribe?
  • By reviewing the results from the commentators as a single body, how do their recommendations stand up to scrutiny by selected academic theory?
  • Finally (and very importantly), what do the DevOps and IT practitioner community make of the KSFs prescribed by the DevOps commentators? Are they relevant, understandable?

The Challenge of Information System Success Measurement

Before we dive into DevOps, I thought I would do a primer on measuring IT success itself.

There is much academic debate on how fruitful Businesses (and IT) are in measuring success. I am not going to dive into this too deeply here, but I do recommend one book in my collection; Benefits Management by Ward & Daniel (2006 or 2012). It is not too big, has a nice IT focus and it’s very accessible.

It goes without saying that accurate success measurement can help an organisation improve its performance and competitiveness, however Ward and Daniel conclude that measurement is not easy and that some organisations can fail to do it well – and others fail to do it at all.

Some of the challenges in success measurement highlighted by academia include:

Perspective

In effect, each individual/stakeholder has a different view on what activity provides benefit. Petter, DeLone and McLean (2012) talk about three stakeholder viewpoints from an IT perspective;

  • Those that fund systems (Leadership)
  • Those that build systems (Developers)
  • Those that use systems (Users).

Each will have their own perspective (and language) on what success should look like.

Key Takeaway: What may excite you as an IT Developer, may not be of interest, or even understood by the CEO.

Digital Transformation Consultation

Benefit Gestation

To summarise, how long will an organisation wait before measuring benefit? There is plenty of evidence to suggest many organisations measure the success of an IT initiative on the date of project delivery – as opposed to after two or three years of usage.

Key Takeaway: At what point does your organisation measure success? Does it go back and revisit projects and initiatives from previous years?

Tangibility

Most formal measurement emanates from the finance department. With a preference towards “hard” financial metrics (e.g. cost, cashflows etc), there is evidence that many organisations shy away from considering intangibles (e.g. agility, self-improvement, collaboration etc)

Key Takeaway: What sort of language do you hear in your organisation around success measurement? Is it what you use?

DevOps (The Briefest of Descriptions)

The research did not seek to extol the virtues of Puppet, Jenkins, Splunk etc (other products are available!) but from an academic perspective had to explain to the reader what DevOps is. There are many definitions and discussions on what DevOps is to be found on the Internet.

If you are interested, I recommend you peruse a few of the respected commentators such as Humble, Kim, DeGrandis, Edwards (look up “CAMS”) and Puppet Labs.

The danger of creating a definition here is that I add to the noise and create anotherdebate!

For now, I will go with a simple definition that DevOps reconfigures the relationships within a supplier delivery chain. The pivotal piece being that this often involves individuals and groups within the same, single organisation performing a subtly different role. They will have to work more closely with each other, exchange and shift control boundaries and learn to trust the output of upstream and downstream colleagues.

I described it in the dissertation as the removal of the wall between two different reward systems. Developers (“we need to get new features out fast!”) are now as equally responsible for Support/Operations (“we protect production at all costs”) as Support/Operations are responsible for fast delivery of code.

DevOps exploits the recent advances in sharable technologies such as Cloud, Network and Data-storage but ultimately, as many commentators agree, the cultural impacts are greater than the technological. It would be possible to have a DevOps site with no new technology, but technology would certainly help!

Reflecting back on the measurement challenges

But if we consider DevOps in the light of the challenges of success measurement, issues begin to emerge.

  • If DevOps has a cultural side, the cultural aspects (empowerment, co-working, learning, cross-skilling) are more likely to be intangible – can we measure these effectively?
  • The patience of a Board could be tested in waiting for success signals to appear – especially if the organisation’s measurement programme is geared around rigid annual financial cycles.
  • Finally will the signals of success be expressed in a language that is meaningful to all recipients?

In Part Two we look at the metrics suggested by the DevOps community and we look at whether they are suitable for different types of stakeholder. We also look at whether the metrics prescribed stand up to academic scrutiny.

Do Take part in the survey!

Whilst the dissertation is complete, the data is still very live. Please take 5 minutes to fill in the DevOps ROI Survey. You can then compare your responses to the general population in later parts of this article.

Use of this article:

By all means print, share and discuss with your colleagues. If you wish to republish any part of it, please add the following reference to your publication.

Linwood, D. (2017) Which metrics assist in DevOps proving its ROI? (With assistance from academia). Part One. [Web-log] Retrieved from https:/uk.linkedin.com/in/davidlinwood

References:

Petter, S., DeLone, W., & McLean, E. R. (2012). The past, present, and future of IS Success. Journal of the Association for Information Systems, 13(5), 341-362.

Ward, J., & Daniel, E. (2006). Benefits Management, John Wiley & Sons: Chichester.

About the Author:

Having spent thirty years in the IT industry in the Banking/Insurance sector, working in IT Development, Support, Operations and Technical Architecture areas, most in Leadership roles, I have been lucky enough to have had a short career-break which has enabled me to complete my studies for an MSc in Management at Ashridge [Hult International] Business School. What you see is a result of that effort.

 

Arrange a Conversation 

Browse

Article by channel:

Read more articles tagged: DevOps, Featured