Hypotheses
Page: https://fernhill.example/checkout. Primary metric: completed purchase rate of checkout page visitors.
| ID | Hypothesis | Evidence quoted | Evidence lines |
|---|---|---|---|
| H1 | If we show the shipping cost before the last step of checkout, then the purchase rate will change, because shoppers say they didn't know shipping cost until the last step. | Obs 2: "I didn't know shipping cost until the last step" (63 of 210 answers mention shipping cost). Obs 1: 41 of 120 reviewed sessions scroll between the shipping options and the order total before leaving. | 2 |
| H2 | If we label the coupon field to say whether gift cards are accepted, then the purchase rate will change, because shoppers ask whether the coupon field takes gift cards and many open it. | Obs 3: 28 tickets in September asked whether the coupon field accepts gift cards. Obs 4: 22% of checkout sessions open the coupon field. | 2 |
| H3 | If we show the running order total next to the shipping options, then the purchase rate will change, because shoppers move back and forth between those two areas before leaving. | Obs 1: 41 of 120 reviewed sessions scroll between the shipping options and the order total before leaving. | 1 |
Each hypothesis names one change and a direction to test. None promises a lift.
Sample size and duration
Inputs: baseline p1 = 3.2% (0.032), MDE = 10% relative, two sided alpha 0.05 (z_a = 1.96), power 0.80 (z_b = 0.8416), 2 arms, 18,000 weekly checkout visitors. Computed by script.
- p2 = 0.032 x (1 + 0.10) = 0.0352
- n per variant = (1.96 + 0.8416)^2 x (0.032 x 0.968 + 0.0352 x 0.9648) / (0.0352 - 0.032)^2 = 49,775 (rounded up)
- Total sample = 49,775 x 2 = 99,550
- Weeks = 99,550 / 18,000 = 5.53, rounded up to 6 weeks
| ID | p1 | p2 | n per variant | Total sample | Weeks |
|---|---|---|---|---|---|
| H1 | 0.032 | 0.0352 | 49,775 | 99,550 | 6 |
| H2 | 0.032 | 0.0352 | 49,775 | 99,550 | 6 |
| H3 | 0.032 | 0.0352 | 49,775 | 99,550 | 6 |
All three hypotheses use the same page, baseline and MDE, so the numbers match. None is over the 8 week limit.
Priority order
Rule: more independent evidence lines first, then fewer weeks needed. If still tied, keep the order the observations were given. All three tests share the checkout page's traffic, so they run one after another.
| Order | ID | Evidence lines | Weeks | Cumulative weeks |
|---|---|---|---|---|
| 1 | H1 | 2 | 6 | 6 |
| 2 | H2 | 2 | 6 | 6 + 6 = 12 |
| 3 | H3 | 1 | 6 | 12 + 6 = 18 |
H1 and H2 tie on evidence and weeks. H1 goes first because its first evidence line (Obs 1) was given before H2's (Obs 3). H3 changes the same area of the page as H1, so it should wait until H1 has been read out.
Test spec for the top hypothesis
- Hypothesis: H1, show the shipping cost before the last step.
- Control: the current checkout, where shipping cost appears at the last step.
- Variant: the shipping cost shown [SET: where, for example next to the cart subtotal at the start of checkout]. Nothing else changes.
- Primary metric: purchases / checkout page visitors.
- Guardrail metric: [SET: for example average order value], so a change in shipping display doesn't hide a drop elsewhere.
- Audience: all checkout page visitors, split 50/50.
- Sample: 49,775 visitors per variant (99,550 total), about 6 weeks at 18,000 a week.
- Stop rule: stop at the computed sample, not when results look good. Read the result once, at the end.
Needs your decision
- Fill in where the shipping cost appears in the variant, and pick the guardrail metric.
- Approve launching H1 in Optimizely or VWO.
- Any change to the MDE or the stop rule after launch needs approval.