back to thedrum.com
Ben Connolly, Head of Digital Engineering at Vodafone UK recently spoke to diginomica about the award-winning DevOps work we have been doing in partnership, enabling first-class digital experiences to be released in record speed.
This article was originally published on diginomica.
For Ben Connolly, Head of Digital Engineering at Vodafone UK, his team’s overriding priority should be to put into production software that delivers solid commercial value to the business - and do so as quickly, as regularly and as smoothly as possible.
Any other activity, he argues, is at best a distraction and at worst an obstacle to meeting that goal. This includes many of the traditional tasks and concerns of a team like his: testing code, provisioning hardware, deployment processes that involve anything more than a single click of a button.
This clear thinking is what has enabled Connolly to take his team on an agile transformation journey over the past two years that has reaped some pretty impressive results.
When he joined Vodafone back in 2017, the digital team, which oversees the mobile phone operator’s web and mobile channels for both consumer and enterprise customers, was using a largely waterfall method and achieving just one software release every two months. Each deployment could involve around 8 hours of downtime, preceded by lengthy manual testing. And his team was highly reliant on the services of third-party consultants to run its development and operations processes.
Today, the team deploys software every day with zero downtime, in a ‘release on demand’ model, by which new functionality is deployed into production and immediately or incrementally available to customers based on their demand for it, regardless of what other pieces of functionality are still awaiting release. Connolly says:
"So in August, for example, we did around 150 deployments over the course of the month - and every one of those releases was immediately out there in production, delivering value back to Vodafone and our customers, without having to rely on the other 149 deployments being in production at the same time."
At the same time, testing is now largely automated, as is infrastructure provisioning and management, and the Vodafone digital team has full ownership of the whole process.
In terms of how this was achieved, Connolly credits DevOps consultants from technical consultancy MMT Digital in helping he and his team to navigate the changes required. These focused on two main areas: first, automating infrastructure build and application deployment: and second, building a robust and sustainable DevOps culture.
In terms of automation, he says, one of the earliest ‘wins’ for the team was the elimination of downtime during deployments, which was achieved by establishing a blue/green deployment on Amazon Web Services (AWS).
This means that two live production environments are in place. New code is deployed on one environment, and traffic can be ‘flipped’ from an existing set of instances running the previous version of code, to the new set of instances running the latest version. Once traffic has been rerouted to the new instances, the existing instances can then be terminated:
"So from a customer’s perspective, there’s no downtime and their journeys are never interrupted, which benefits them and the business. That was a great success, but kind of an obvious first step - but from there, we’ve matured to a place where we’ve achieved the genuine automation of our infrastructure and one-click deployment. This has involved a great many different tools - an end-to-end suite of tooling, recommended by MMT Digital, that has fuelled not just our ability to release software quickly, but also helped us to build the culture that we wanted to achieve as well. When I think of the tooling decisions made, I can’t think of a single one that I regret."
These tools include AWS Cloudformation and Terraform for the infrastructure as code capabilities that enable new environments to be provisioned in less than 30 minutes: Ansible, for configuration management; Pagerduty, for routing service alerts and failures to developers so they can manage incidents with their own in-production services; and DataDog, for monitoring and alerting.
Also in the mix is Microsoft Azure DevOps, which provides a ‘single source of truth’, including tools for agile planning and tracking, code storage, version control repositories and deployment pipelines. Docker is used for microservices orchestration and Appdynamics for application performance monitoring. Slack, meanwhile, is used as the main communications channel between development teams - more of which shortly.
In terms of making changes in ways of working, Connolly and his team have adopted the Scaled Agile Framework (SAFe), an approach popular with large enterprises. Of this, Connolly says:
"I think in the early days, we kind of swallowed the Bible a little. Don’t get me wrong - the framework was incredibly useful in helping us get started in terms of overlaying onto the organization the processes and practices we were previously completely lacking. But the risk lies in valuing adherence to any framework over business results. That happened here, but it didn’t take us too long to figure that out and get back to putting business value first. What I’m trying to say is that completing a successful sprint isn’t, in itself, a good measure of success. What matters is the value in terms of a successful customer experience that the sprint delivers."
Slack, meanwhile, bridges these two worlds of automated infrastructure and humans working along DevOps lines: environments can now be created by team members using Slack commands, for example. Software can be deployed through the messaging system. When a bug is logged, it will be logged in a bug system, but a notification will pop up in Slack for the relevant engineer, linking them directly back to that bug. Says Connolly:
"From a leadership perspective, this is great. I can see everything that’s going on, at a glance, by subscribing to the relevant channels. That shows me all the communications that are happening around an issue that we’re facing or an initiative that we’re launching, so I rely a lot less on pre-prepared, choreographed reporting, after the fact, in which bad news might be conveniently hidden away or reshaped. It’s made everything more transparent, which is genuinely fundamental to any agile journey.
"It’s not been easy. At times, it’s felt like an uphill battle. But by bringing people along and demonstrating what these changes are achieving in terms of value to the business and to our customers, I feel like we have a real head of steam now, there’s real momentum. We’re a more responsive organization and much more limber in our approach to delivering products and experiences and the results we are reaping speak for themselves."
