Thoughts from a Thirsty Bear

As AI adoption continues apace, one good thing coming out of this second mass experiment of the decade (the first being Work From Home, caused by Covid) is  data . As more and more companies adopt AI in the software development workflow, they are generating huge amounts of information about its effectiveness. The folks at CircleCI are in a unique position to collect the data across millions (28 million to be precise) of delivery pipelines they host. They have published a summary of their findings  here . So what do 28 million data points tell us about using AI in software delivery? It's..."interesting". TL;DR - AI's effect on branches is noticeable. It increases throughput of branches for median teams significantly (15%), but this increase never makes it to production; they are never released. The production release branch throughput degrades by ~7% for the average team - AI adoption slows down teams by quite a lot. As expected, the AI-induced pressure of generating m...

 We are certainly living in interesting times right now. Specifically here I am talking about the rise of "Artificial Intelligence" (or, more correctly, 'Large Language Models", AKA 'LLMs') in software development. Whether you believe these tools are useful, or will lead to the demise of the human race¹, there is no doubt they are surrounded by the biggest hype cycle that anyone has seen before. There is an unprecedented (many are now saying unsustainable) amount of investment happening in this area. But any investment requires a return which means marketing and sales are working overtime to bring in new paying customers. Unfortunately as a result of these intensive sales campaigns there are companies - many of whom should know better - becoming ensnared in the unfulfilled promises being made by unscrupulous companies, and are " drinking the Kool Aid ". However, once the contracts are agreed by senior management, and the (not insignificant) bills be...

"A little learning is a dang'rous thing; Drink deep or taste not the Pierian spring: There shallow drafts intoxicate the brain, And drinking largely sobers us again" - from " An Essay on Criticism, Part II ", Alexander Pope Those who know me, or have read my profile online, know that I teach teams how to develop software using iterative tools and techniques. These require rewiring some of the traditional, waterfall-style thinking and approaches, and yes, they can be a culture shock. In order to warn my trainees I always go through a specific explanation with them. It kind of goes like this: Many folks expect learning to be straight line. You do a training course, or read a book for something, then you use it.  Unfortunately this is the South Park Underpants Gnomes model of learning. It leaves out the important part. Learning does not work in a straight line. It's messy. It's unpredictable. You will make mistakes. Go down blind intellectual alleys. Get co...

 ...and they can generate bad code far faster than any human... In case you've missed it, there's is a new buzzword going around. Artificial Intelligence, aka "AI". If you believe the hype, these clever little learning programs are the solution to everything, from creating fine art to curing cancer, to writing software. They vacuum in existing data (the 'training set'), then use this knowledge dump to decide how to solve problems they are given. " It writes code? Great! ", I can hear you say. " I can use this to write my code faster! ". Certainly this is a comment I am hearing from managers, and also inexperienced developers. Not so fast, folks... Several experienced developers, including myself, have been poking around these code generation tools to see if they live up to the hype. I can assure you they don't. I am not an AI expert, but I strongly suspect that this is a function of how these tools work. Think about it - the learning ne...