Design Guidelines for the best Improvement

First let’s start with a good case study. We are now closing a strong Q2 for our clients where we averaged a reduction of 27% in the overall defect inflow on all projects. This is across domains (Automotive, Retail, Healthcare) and across technologies. This is a lot of free capacity for those clients. Some of them have decided to save costs and downsize the teams by 25%. Some have decided to deliver 25% more content in the next iteration with the same teams. In both cases, the improvements have immediately created a benefit to the business, in other words, progress.

Let’s try to create an acceptance criteria for an improvement proposal, assume you need to approve the time, budget (or both) of an improvement or you are considering starting one yourself. Here is the acceptance criteria: 

• Will this initiative improve time to market?

  • By how much?

  • How will we measure it?

  • When will we feel it?

• Will this initiative improve R&D costs?

  • By how much?

  • How will we measure it?

  • When will we feel it?

• Will this initiative improve customer satisfaction?

  • By how much?

  • How will we measure it?

  • When will we feel it? 

If the initiative owner can’t explain how the initiative improves one of the main three pillars: time to market, R&D costs or customer satisfaction then they should be allowed to spend time or budget on that initiative. If they can’t provide a clear measurement that translates to actual budget freed up, resources freed up etc. they should be not allowed to invest in this initiative. It will most likely only produce local benefits, will not improve the overall system and the time is better spent somewhere else.

To make improvements pay for themselves, the best course of action is to start by focusing on a problem that costs money today. If you can find one (usually not very hard) and you can eliminate or minimize it, it should be very easy to show how the improvement pays for itself.

Let’s take two examples from two of the farthest areas of the SDLC, customer satisfaction and software requirements and see how it looks when an improvement pays for itself.

Starting with customer satisfaction, look at the stream of issues coming from your customers in the form of defects, tickets, customer complaints, customer escalations etc. There are many common models to quantify the cost of each customer complaint. Depending on your industry the model can be different but in any case the numbers vary between $10k to $100k per customer escalation. A good initiative would be, for example, to analyze this stream of customer issues and start adding to your automated regression suite a test case designed to catch the exact issues reported by your customers. You don’t even need to eliminate the root cause, just focus on catching those issues before they reach the customer. In every defect cost model in existence the cost of catching defects in-house, even at the last stage is 10X lower than catching them at the customer site. This makes sense if you take into account the cost of an escalation including your reputation costs, future business costs and the sheer time invested in calming down the customer while debugging on site and providing a hotfix. If your initiative is able to “move” the detection of those customer issues upstream from the customer site to your very last phase of testing, you would have saved a lot of money. 

Let’s take some numbers: assume 50 customer escalations per year and assume $10k per escalation, this is now costing you 50 X $10k = $500k per year.

Assume you invest 1 full time person in automating every customer incident into your full regression suite and it takes them a year to add those 50 test cases into the suite.

There are always new customer issues, so let’s assume this initiative is only able to reduce 30% of the current escalations, which translates into a savings of $500k X 30% = $150k per year.

You’ve invested the cost of a full time person, perhaps $50k. 

So, for the first year alone your ROI is $150k / $50k = X3. for the second year you save those $150k without any investment. Of course, you can (and should) continue this practice forever and as you can see this is paying for itself. This is a good example of how it’s done. You need to adapt the numbers to your situation of course but the principle applies.

Now let’s switch to software requirements. Look at the stream of issues created against the existing requirements. Look at the rework caused by each. A mistake in a software requirement is usually the second most costly issue across the SDLC (customer escalation is the worst) because it means something was defined, designed, implemented, tested and only then found to be wrong. Measure the amount of wrong requirements, which is called your requirement escape ratio. Each escape is costing you something, you can use any existing model to quantify, typically this can be between $1k to $5k. Once the cost is established, we can design the improvements. Let’s take some numbers:

Assume a defect inflow of 1,000 defects per year and about 20% requirement escape ratio

This would be 1,000 X 20% = 200 software requirement defects per year

The cost for those would be $1k X 200 = $200k per year

Now implement a basic improvement, add a review phase for each requirement approval with a forum that consists of the most senior architect and most senior test person in the organization. Each requirement review now costs you $1k more because of the time spent there, and there are 200 requirements so the cost is $1k X 200 = $200k

The improvement is meant to reduce the number of requirement escapes from 20% to 15% (which we obviously continuously measure). This means the improvement is going to save us:

20% – 15% = 5% of escapes

5% of escapes X $1k cost of escape X 200 escapes per year = $10k

So this improvement is going to save us $10k and is going to cost $200k – a very bad investment, this improvement should not be approved.