Many teams use AI at various phases of the SDLC such as testing, generating requirements, doing code review, writing code, writing tests etc.
It is becoming clear that it is very hard to prove those investments are yielding actual progress or ROI. Let’s look at the reasons
When does Investment actually lead to Progress?
You can invest in many different initiatives and you probably do. Building more test labs, buying more tokens for your developers, replacing your configuration management tool, replacing the documentation engine, training the team on a new working methodology. In fact, most engineering leaders invest up to 20% of their budget on various improvement initiatives. In these days, many of those investments are around deploying AI solutions across the SDLC
Then comes the question of measuring the ROI, which is similar to asking “are we making progress”. When asking that, most engineering leaders mean one of three things:
- Are we delivering faster features to the market?
- Are we delivering higher quality, maximizing customer happiness?
- Are we reducing cost, increasing margin
Many times it’s a mix of all the above
So, does investing in improving code review with AI or documentation creation with AI lead to progress on those metrics? hard to say
It’s important to understand that investing in an area of the SDLC will (most likely) lead to an improvement there. It is also important to understand that improvement does not mean progress and definitely does not mean progress towards the above mentioned goals
it is only progress when you are improving the critical path to success. In other words, it is only progress when you’re improving the right things
What needs to be done and if often missed is an overall assessment of how the SDLC is used in a team and where the bottlenecks are. You need an understanding of which area in the SDLC implementation in your team is hurting the goals you are trying to achieve. Improving anything else will not lead to progress.
A good example is code generation with AI (think Claude). Sure, you can double, triple the speed at which code gets created, but very often, code creation is not the bottleneck to happier customers. Very often it’s the proper requirement elicitation, the proper testing, stabilization or even customer training. Without realizing that, you can pour more and more tokens into your development team, have more and more code and still achieve no progress towards your goals
And therefore, to achieve progress, find out what’s holding you back and invest in it heavily. This is a dynamic process due to the nature of bottlenecks, you resolve one, you get a new one appearing. This process needs to be iterative
As a rule of thumb ask yourself: “how will this investment support one of the main goals?” if there is no clear answer, it probably means this investment represents a local improvement over real progress
The Smart Way
AI Can do a lot to improve your team’s performance. We’ve seen great examples of it
The answer is, to analyze your entire SDLC and invest heavily, and only, in the weakest link in the lifecycle. And, do this repeatedly. This will ensure you’re maximizing your investment budget and improving at the best pace possible for your investment budget.
Once you do that, start experimenting and enjoy the ride!
