Workflow to Control any AI Agent

Here’s the workflow you should follow:

1. Identify a step in the SDLC you’d like to improve with deploying AI
   1. Optional: qualify, is this step an actual bottleneck in your SDLC
2. Create a measurable quality gate for that step
3. Ensure one of the next steps in the SDLC has a human review guardrail
4. Deploy AI in the step alongside humans
5. Benchmark humans and AI on the same quality gate
6. Benchmark humans and AI on escapes caught at your guardrail
7. Once the AI outperforms humans, stop using humans in that step
8. Repeat for another phase in the SDLC

Let’s take a few examples on how to do that in practice, we’ll look at code review, code generation, and requirement generation

Code Review

Let’s assume this is the step in the SDLC we’d like to replace with an AI agent because we believe it is not being done effectively today (step 1 and step 1.a)

For our measurable quality gate we will use code review comments (step 2)

For our guardrail we will use code review escapes, which is how many defects are found in testing which could have been identified in the code review phase (step 3)

We start using AI to perform code review alongside humans performing code review (step 4)

We manually review code review comments made by the AI agent and made by humans and evaluate their performance (step 5)

We measure code review escapes in testing (step 6)

Once the results of step 5 and 6 show code review is done better by AI, we replace humans in this step by fully deployed AI (step 7)

Let’s head over to do this for Code Generation (step 8)

Code Generation

Let’s now assume this is the step in the SDLC we’d like to replace. We’re impressed by the pace of code generation by AI (step 1 and step 1.a)

For our measurable quality gate we will use code quality metrics like static analysis and code maintainability (step 2)

For our guardrail we will use code review (human) comments and feedback (step 3)

We start using AI for code generation alongside humans performing the same (step 4)

We manually review code quality indicators for human generated code and AI generated code and evaluate their performance (step 5)

We measure code review findings, comments density, comments severity, number of iterations (step 6)

Once the results of step 5 and 6 show code generation is done better by AI, we replace humans in this step by fully deployed AI (step 7)

Let’s head over to do this for Requirement Generation (step 8)

Requirement Generation

Finally, let’s do this for requirement generation (step 1) because we believe current requirements are weak and AI can do the research better and come up with solid requirements (step 1.a)

For our measurable quality gate we will use requirements review against our checklist (step 2)

For our guardrail will use requirement review escapes, which is how many defects are found in testing which associated with issues in requirements like incompleteness or wrong requirements (step 3)

We start using AI for requirement generation alongside humans performing the same (step 4)

We manually review requirements for human generated requirements and AI generated requirements and evaluate their performance (step 5)

We measure requirement review escapes in testing (step 6)

Once the results of step 5 and 6 show requirement generation is done better by AI, we replace humans in this step by fully deployed AI (step 7)