SAPPHIRENow 2019 – Coke’s secret IT formula revealed – ditch waterfall for DevOps

Coca-Cola might be best known for its secret formula, giving us a red-suited Santa Claus, and being a life saver for fizzy-drinks-loving dieters worldwide. But at Sapphire this week, the organization revealed a different secret recipe: how to manage a disparate technology infrastructure using DevOps.

The impetus for Coke One North America (CONA) Services – the firm’s IT platform – to turn to DevOps was being able to push out new and enhanced features to its bottling partners more regularly and quickly.

Internally, CONA was happy with its technology landscape, which is built around SAP technology for everything from finance and HR to reporting and pricing promotions. At CONA’s core is the SAP ERP Central Component (ECC), along with its GRC Solution Manager, the SAP portal, some Fiori applications, SAP Concur, SuccessFactors and a large business intelligence Hana environment.

But despite this investment in its IT platform, getting the latest and best technology out to its bottlers – Coca-Cola sends out its syrup, and then its bottling partners add the carbonated water, bottle it up and manage the distribution – was proving tricky.

Coca-Cola has just 12 bottling partners that account for about 95% of volume in North America. Stats cited at the presentation included that there are “just under” 81,000 active employees using the CONA system, who are responsible for almost two million ship-to customers, 43 million deliveries and 5.3 million equipment service orders.

On the old Waterfall-based way of working, everything was just taking too long. Brian Toms, director PMO & Enablement, CONA Services explained:

I think we’re very much like a lot of companies that have been around for a while – we were very Waterfall centric. So as we started to deploy the common solution to all these bottlers, it wasn’t just 12 deployments. We had to roll it out to the territory all across the United States and across Canada as well. So it took us a couple of years to do all these deployments.

While we were doing the deployments, we were also building new capability into the solution. And we’re doing that by delivering two major releases every year. So not all that uncommon for most SAP shops to have a couple of big releases every year, one technical in nature and the other really heavily focused on functionality.

As the bottlers were trying to absorb these big deployments, at the same time CONA was trying to build out new capability. Result: everyone is pressed for time because as the partners are doing the deployments, CONA is also doing its own deployments and trying to build in new features. Toms added:

So you really end up squeezing things and by the time you get to testing and you get back in front of your customer – in our case, the bottlers – what we may have built may not necessarily have been exactly what they were looking for. Now you start getting – what if we did this, or I forgot to tell you that. And you get scope creep at the end.

CONA knew this model wouldn’t work going forward, so the focus became how to very quickly start delivering things faster and of higher quality. As the bottling business is a very low-margin business, it also needed to make sure the costs were managed appropriately.

The firm realised that an Agile DevOps model would suit its needs (it’s worth noting here Coca-Cola’s definition of DevOps, as there are various ones out there – as the firm sees it, DevOps is a set of practices and cultural changes supported by tools, which automates as much as possible). Toms added:

You’re basically breaking things down into smaller increments. I’m constantly getting in front of the customer. At the end of the day, you’re delivering really what the customer wants from you.

So you may be able to start giving value to your customer much quicker. Even though they’re not getting 100% of what they want, they get an MVP [minimum viable product] that they can start to utilize, they can start to realize value and give you feedback on that. That may influence the rest of what you’re building as well.

CONA has gone from having two major annual releases to having “no releases at all”. Instead, everything now moves on a daily basis. The firm has a specific criteria for technology releases, and once something is ready to go, it goes through a review, the box is checked and it goes out the next day. Toms added:

If you don’t have all your ducks in a row, maybe you’re waiting another day. But it’s much faster than what we were doing before to be able to get things into production.

And because we’re breaking things down into much smaller increments instead of these big releases – for all of you that have dealt with big SAP releases, you move to release it and everybody holds their breath waiting to see what’s going to happen. And then if something goes wrong, how do you figure out what actually happened? What broke it? If you’re moving things in much smaller increments, it’s highly likely you’ll be able to immediately identify if something’s broken.

One of the key priorities for CONA under the new system is try to make sure it’s delivering quality on time and within budget. Initially, the firm started out with project sprints at 15 days, but through continuous improvement it has modified that to four weeks. Feedback from the teams highlighted that 15 days was too fast to be able to build and continuously plan and manage the backlog.

For other firms considering a move to a DevOps model, here are CONA’s key takeaways.

Change management

Hemant Kochhar, Director of Development and Integration at CONA Services, said this managing change is the major factor, as this is a big cultural shift and not an easy one. He said:

It’s hard and everybody in the organization must be on board. So it takes a while to get there. I think we are still working through that process, it’s not a quick thing. You have to work through it and plan for it.

CONA took the approach of piloting the project system in two different product areas that would get the highest value right away. It engaged a consulting organization to train up team members on Agile, and it provided training for the bottlers so they would understand the approach and expectations.

Backlog

For CONA, the goal was to have a single backlog – workload of IT jobs. Having a single list of what needs to be delivered, whether that’s setting up a new printer or a larger piece of work requested by a bottler, is vital. Toms said:

Some of the things that come into our backlog have built-in SLAs, so we have to prioritize them accordingly. For a change request on the front end, we’ll do a quick evaluation of it and then we commit to a date when we’re going to deliver that to the bottlers, because we need to be able to set an expectation, not only of the date, but also of the cost.

Define done

The definition of ready and done have been critical for the project. Toms explained:

If nothing else, having a definition of done for each of your increments is gold. To be clear on how do I know when I’m done with this thing that I’m doing. Also for us, one of the more important things is to understand how it’s going to be tested by the bottlers because we may have a perspective on how we’re going to test it, but we also want their perspective of how they’re going to test it because that may influence requirements in a way that we weren’t thinking about. A definition of done is critically important.

Automate

CONA wants to automate everything as much as possible. Kochhar said:

If you’re know you’re going to be doing things more than two or three times, find a way to automate. If you can do that, that will help speed up and make things much more consistent.

Measure

Meaningful KPIs give visibility to the right things – issues as well as successes. Toms said:

Visibility is probably one of the biggest benefits that we’ve been getting out of this. Because of the daily stand-up meetings that we have as well as the measures and the metrics, it’s very visible what’s going on at any given point in time. And if anyone has run into any blockers, it’s something that we can immediately address to keep progress.

In retrospect

For CONA, this work is a constant process that will never be done. Instead, it’s an opportunity to continue to improve. Toms explained:

As you roll this out, it’s not a once and done, it’s ongoing. At the end of each sprint, when you do the retrospective, the light bulb goes off that we can change it now. So at most, you’re doing something for three or four weeks, whatever your sprint or your iterations are, and then you adjust. Learn from your mistakes, take what’s good and move on.