How to Measure Requirement Correctness?

The first step is to get a good understanding of injection points. Every time a problem is detected, we can look back in time or in the process that led to this problem and understand where the problem was created (injected). In the software world, problems arrive in the form of bugs or defects and the good news is that for each defect there could be only three injection points. It could have been injected (created) as a result of a misleading, incorrect or incomplete requirement. It could have been injected (created) as a result of inconsistent, incorrect or incomplete design or architecture and finally, it could have been injected (created) as a result of a problem in implementation (coding). Those are the only options. 

Some teams confuse the detection point with the injection point. The detection point is where the defect was found. If could be found as early as a design review or as late as a customer escalation. If could be found in the nightly continuous integration run or in the manual pre-release regression testing run. Detection points are many and could be as many as there are test layers and activities. Regardless of the detection point, the injection point is always either requirements, design or implementation.

Once we understand injection points, we can look at a subset of the defects opened in the last iteration. We can also look at all of them if we have time or if the number is not too high. As part of your retrospective, perform the following exercise. For each of the sample set defects establish what is the injection point, you can use your experts, a team of experts or representatives from each discipline. The outcome of this exercise is a determination for each of the defects of its injection point. Remember, it can only be either requirements, design or implementation. 

Next up, you should calculate the percentage of defects with injection point set at requirements out of the total number of defects analyzed in this exercise. This is your requirement escape ratio. The percentage of issues that were injected in the requirement phase and were caught too late. You know they were caught too late because if they were caught as part of the quality gate at the end of the requirement creation phase, they would have been addressed without a defect being recorded. Similarly, you can (and should) calculate the design escape ratio and implementation escape ratio. Those three ratios should sum up to 100%

Now you know how big of a problem requirements are. If your escape ratio is larger than 15%-20%, you definitely need to look at your requirement definition process. Every defect with an injection point in the requirement area has a very high cost associated with it. It is probably one of the most expensive defects to handle. And so, the requirement escape ratio should be kept low, typically at around 10%. Getting it lower than that usually involves a lot of effort which starts being above the point of diminishing returns. 

You now have the same visibility into your design escape ratio and can work to improve it as well. Again, it shouldn’t be more than 15% of your issues with 10% being a good place to be.

Most issues, obviously, will be injected during the implementation phase. Here you should not try to drive this number down as driving it down means driving the requirement and design escape numbers up (it’s a zero sum game). Here the focus should be on early detection. Here we should measure the code review escape ratio, the number of issues caught by the merge gate and number of issues caught in the nightly continuous runs. Those numbers should go down not as a percentage but as absolute numbers.

And there you have it. Now you absolutely know whether requirements are the problem in your organization and if you’re a requirement owner, you can proudly say “our requirements are stable more than 95% of the time” and you can prove it! That’s such a strong position to be in…