When does an Improvement actually Matter?

“If you’re not worried, I should be worried” is an old time saying that most enterprise employees have long internalized. Very few people feel comfortable saying “we are perfect, there is not so much more that we can do”. They know that if they’re not worried, you, their manager will become worried. 

You, an as engineering leader, are always under pressure to deliver one or more of the following three:

1. More products, and faster, to the market
2. Higher quality, higher customer satisfaction 
3. Lower costs, better margins (deliver more with less)

Your teams, knowing they can’t be overly satisfied with the current state will always have some improvement roadmap, some plan that they’re working on that will make things better. Those improvement plans are everywhere now and they serve as great safety net. Can’t be to unhappy with us, we’re improving…

Now, of course, there’s always limited budget, limited time and limited resources to invest in those improvements and so there needs to be some criteria to select, some criteria to choose which ones to prioritize and which to discard.

The decision factor is simple to state and not so simple to execute. Any improvement, any investment, should clearly states how it ties to one of the three objectives listed above. If the initiative can’t directly link to a business outcome, most probably, it’s going to be a waste of time. Let’s take a few examples:

Example #1: A test architect wants to automate the entire test regression suite – sounds like a good initiative, who can oppose to automation. let’s tie this to a business result.

Test automation by itself is not a goal, it is an enabler to running tests often. Being able to run the regression suite daily and identify blocker issues early is certainly less costly than finding them in the integration rounds. When we query our last year’s results we can see we found more than 100 blocker issues in integration. more than 80% of those issues are part of the regression suite. So by automating the regression suite we can eliminate 80% of the blocker issues, certainly contributing to objective #3. In this environment a blocker issues found in the integration round costs $1,500 and a blocker issue found over night costs “only” $500. Therefore, by moving 80 integration blockers to the nightly test run, we have saved the organization 80 X ($1,500 – $500) = $80,000

Automating the regression suite will cost $40,000 in labor. We can now see that for a single year the ROI will be greater than X2 for this initiative. This is a good investment

Example #2: A dev architect wants to refactor a critical, failing component – sounds like a good initiative, who can oppose to refactoring. let’s tie this to a business result.

Refactoring is not the goal. this component has caused 10 customer escalations in the last year, as become evident by the roof cause analysis we did to all customer escalations. Each customer escalation costs us $10,000 in escalation calls, work over weekends, reputation costs etc. It is estimated that this refactoring will eliminate 70% of the customer escalations. Which means, we expect to save 70% X 10 X $10,000 = $70,000 with this initiative.

The investment is a single sprint of labor, which will be less than $20,000, this gives us an ROI of >X3 and make this a good investment. It directly contributes to objective #2.

Note that, with those examples, not only do we quantify the improvements, we tie them to business performance, but we also create a mechanism that allows us to measure the initiative success. In the latter example, if the customer escalations for this component do not go down after the refactoring, we can deep dive, understand the root cause and use the outcome to better justify further investments.