The bitter pill

As AI adoption continues apace, one good thing coming out of this second mass experiment of the decade (the first being Work From Home, caused by Covid) is data. As more and more companies adopt AI in the software development workflow, they are generating huge amounts of information about its effectiveness. The folks at CircleCI are in a unique position to collect the data across millions (28 million to be precise) of delivery pipelines they host. They have published a summary of their findings here.

So what do 28 million data points tell us about using AI in software delivery? It's..."interesting".

TL;DR - AI's effect on branches is noticeable. It increases throughput of branches for median teams significantly (15%), but this increase never makes it to production; they are never released. The production release branch throughput degrades by ~7% for the average team - AI adoption slows down teams by quite a lot. As expected, the AI-induced pressure of generating more code creates bigger problems at the review-and-release stage. Which indicates that AI simply is not worth the money for most teams unless the company wants to get even worse at delivering value to their customers (which is usually business suicide).

BUT...the data also reveals something that I was not expecting.

For the top 5% of teams, AI can provide a whopping 26% improvement in flow through the main branch. While this may or may not indicate more value delivery (the study isn't clear), it is still a significant increase of flow into production. However this effect tails off fast - by the top 10% it's down to 1% (so again, probably not worth the AI token bill). What's the difference with the top 5% elite? It is a combination of technical skills, team psychology, and a streamlined flow of value to the customer. 

This should raise an uncomfortable truth for teams and managers everywhere: Are we really in the top 5% of team performance? From my experience over the years in the software industry, 95% of teams will say (and probably truly believe) they are right there in the top 5%, which makes absolutely no sense whatsoever! 

It's a bitter pill to swallow, but as the poster says, "Not everyone will get to be an astronaut when they grow up". 95% of teams out there do not possess the fundamental software delivery skills to get any benefit from AI assistance. The evidence seems to show that AI will make most of them even worse at delivering value. 

So how does a team tell if it's truly elite? What does a "top 5% team" look like exactly? I have already written about what makes a high performing team tick relatively recently (spoiler: it's not AI), but let's dig a bit deeper to put some fundamentals in place. 

Firstly, and most recently there is the work of Professor Nicole Forsgren and Jez Humble, backed by DORA. In 2018 they published a groundbreaking study in the book Accelerate. They found the most value being delivered came from teams that could do on-demand deployments, less than an hour lead time for change (code commit to running in production), less than an hour to recover from a bad deploy, and a change failure rate (ie. how often a production roll back was needed) of 0-15%. This is the target 'landing zone' for teams before they should even be considering using AI assistance. 

Which raises the question of how? How do teams reach this level of performance? I believe there are strong clues in Jez Humble and Dave Farley's 2010 book, Continuous Delivery. They present a capability model for teams that I still find useful today for assessing where a team is on their journey.

You can see there is a lot of emphasis on automation, collecting and acting on metrics, all coupled with a strong focus on fast feedback, constant improvement and also team responsibility to improve all the time. Good, old fashioned software engineering that has been enabling high performing teams deliver stable software faster for decades. Without this basic capability, new tools - like AI - will simply fail. They get in the way because they optimise the wrong thing.

Yet even this is not enough. My experience with high performing teams indicates that there are team knowledge and behaviours underpinning performance. It starts with a deep understanding of software structure, and the need for adaptable, malleable architecture and design - something aided by skills in Test Driven Development. It is also no coincidence that using Trunk Based Development has a significant impact as well. On the human level, I have always found a deep level of humility and willingness to learn in high performing teams. Nobody is defined by their code, and everybody is happy to learn from each other. Collaboration is everywhere in these teams - many pair and ensemble program together as a matter of course. Work is visible and accessible, problems are swarmed on as a team. High performing teams also have fun! There is always a certain buzz around teams that truly collaborate openly and freely.

So, given this standard, is your team really in the top 5%? Or do you have work to do before reaching for that AI to put the icing on top?

Finally there will be people out there saying that AI will allow us to ignore these traditional engineering disciplines and simply deliver software without basic skills. The numbers are stacking up against that view. I would also point out the extreme lack of challenger companies coming through with their great AI-generated applications. We have been studying the fundamentals of software delivery for decades now, and they do not change. Don't fall for the AI zombie mind virus - get back to basics, get those sorted, and once you are delivering on demand within an hour of a commit, then try AI tools and see if they help. You could be in for a surprise.

===

Want help getting your teams into that top 5%? Want your teams deliver better, sooner? Want to adopt AI successfully by putting the basics in place first? Get in touch and let's talk. I have raised the game of many teams over the years, using combinations of classroom-based training, acting as a C-level/team-level advisor, and acting as a player-coach embedded in teams.