Colabmo development course A-P02

Learning objective: Write a falsifiable hypothesis and a practical evidence plan before testing.

Test the riskiest assumption, not merely the easiest feature to demonstrate. “People liked the prototype” tells you little if the design depends on reliable exception routing. Write a hypothesis connecting a mechanism to an expected outcome for a defined population.

For example: “For standard requests with complete inputs, allowing independent checks to start together will reduce elapsed time without increasing incorrect dispositions or total staff effort.” This contains several measurable claims. Define complete inputs, standard requests, elapsed time, incorrect disposition and effort before collecting results.

A test plan specifies eligible cases, exclusions, assignment or comparison method, observation period, sample rationale, measures, permissions, stop conditions and decision rules. Use a simulation or historical replay first when live consequences would be difficult to reverse. A replay tests some mechanics, but cannot establish real customer behavior or adoption.

If feasible, use a concurrent comparison with comparable cases. If using before-and-after data, record differences in volume, complexity, staffing and season. Do not attribute every change to the redesign. NIST’s experimental-design guidance explains that designs should be selected against objectives and factors; its catalogue includes randomization, blocking and confounding. Seek appropriate statistical help when a decision requires more than a small learning trial.

Set the decision rules before results arrive. A pilot may justify revise, stop, or expand the test. It rarely establishes universal effectiveness. Record adverse outcomes, missing observations and excluded cases, including whether exclusions could make the result look better.

Separate the test’s success threshold from the evidence strength needed for wider adoption. Ten successful cases may show that a mechanism is possible. They provide limited assurance about rare failure modes, unusual requests or sustained performance. State what remains unknown and how the next stage will address it.

Practice

Use W8 for one W7 uncertainty. Specify what result would make you abandon or change the design. Include one quality guardrail, one workload guardrail, an exception case, a comparison and an explicit decision owner.

Check your reasoning

A trial reduced elapsed time but required a specialist to rescue every case. Did the design succeed?