Use this before selecting a pilot and again when reviewing the result. It is a planning framework, not a promise of savings or a substitute for evaluating your own workflow.
Describe one task
Workflow: What starts the task, what input does it use, and what counts as complete?
Owner: Who can judge the output, approve changes, and decide whether the pilot continues?
Current baseline: How much time does a representative task take, including checking, correcting, and handling exceptions? What quality does the current process achieve?
Pilot boundary: What is included? What is explicitly left to a person or another system?
Define the conditions for a useful result
- Quality: What makes an output acceptable? Which errors are unacceptable?
- Time: How will you compare total effort, including review and correction?
- Cost: Which provider, integration, and operating costs need to be counted?
- Exceptions: When must a person take over, and what context will they receive?
- Data and access: Which inputs and tools may the pilot use? Who approves that use?
- Operation: Who owns monitoring, provider changes, and the route back to the previous process?
Agree acceptance criteria before building. The appropriate thresholds depend on the task and its consequences.
Test more than the demonstration
Use a representative set of ordinary, incomplete, ambiguous, and failed inputs. Keep examples separate from tuning work where practical so the review can expose unfamiliar cases.
For each test, record the input category, expected result, actual result, review time, correction time, and any exception. Include provider or tool failures in the operating review.
For an engineering workflow, also inspect the effect on code review, tests, rework, and delivery. More generated code is not sufficient evidence of better delivery. DORA’s research on effective AI use discusses this measurement problem and the effort required to verify outputs.
Make the next decision explicit
Expand: The pilot meets the agreed criteria, responsibilities are clear, and the wider scope can be operated.
Revise: The result is promising, but a specific gap needs another bounded iteration. Name the change and the evidence required.
Stop: The result does not justify the review effort, cost, risk, or integration work. Record what you learned and return to the previous process.
Whatever the decision, retain the evaluation notes, limitations, owner, and next review point. AI Enablement & Automation can help define the pilot and work through that evaluation with your team.