How the programme is measured and what each journey tests: rulings first, then the roadmap, then the headline test per journey. No holdout, no causal read. Roadmap-of-record owner: Owner of the T&L roadmap-of-record = OPEN (Data Strategist role unfilled), Sheran to name
These five govern every journey's test design. Three are ruled, two are not. Read the provenance column before quoting any of them. Reconciled against the ten journey review packs, 27 Jul 2026; each pack remains the current source for its own journey.
| Rule | What it means | Status & provenance |
|---|---|---|
| 1 · Attributed, not incremental | Until the long-term holdout is live, every figure in the programme is attributed. Last-touch attribution is not incrementality, and no attributed number is presented as a result. One exception: the day-before email family (GRW-16) reads +16.9% / +16.3% incremental vs control (confirmed, CRM estate, Apr to Jun 2026). No other moment may borrow that figure. | RULED · Sheran, 9 Jul 2026: the long-term-holdout commitment, with attributed-not-incremental labelling until it is live. The holdout STRUCTURE is still open at decision DL-018 |
| 2 · One headline test per journey release | Each journey release carries exactly one headline test. Everything else is a phase-2 row, gated on the headline read. | RULED · Sheran, 14 to 20 Jul 2026, the test-discipline rule. Some summaries drop the word release; a journey can have several releases, so the rule is per release, not per journey. Cite it by name: no single decision-log id covers the rule itself, and the per-journey headline tests are logged separately (DL-041 Journey 1, DL-046 Journey 2, DL-062 Journey 3, DL-074 Journey 4, DL-097 Journey 6, DL-117 Journey 10). |
| 3 · The double read: journey lift and model targeting are read separately | Where a propensity model both selects and times the sends, a single holdout confounds the model found people who were coming back anyway with the journey brought them back. Two reads, both required: (1) journey lift, a holdout within each propensity band, treated vs untreated; (2) targeting value, propensity-keyed waves vs a batch arm, each with its own holdout. No holdout, no causal claim. The cautionary precedent is the blanket recovery read at −11.8% bids (confirmed, Analytics). | RULED for Journey 7 only · decision DL-103, closed as the working assumption by the Journey 7 council, 20 Jul 2026. Owner: Analytics (Ivi) designs and owns, Sheran approves. The general separation rule is carried in every pack; portfolio-wide adoption is asserted in the executive recommendation but no decision record extends DL-103 beyond Journey 7. TBC, Sheran. |
| 4 · Holdout structure | The working assumption the packs design to: a small global always-out control for programme-level reads, plus canvas-level holdouts on the moments carrying a £ claim. Analytics sizes both layers before any test is wired. | OPEN · decision DL-018, Ivi (Analytics) recommends, Sheran ratifies, due 31 Jul 2026. Not settled: a late structure decision delays every powered read in the quarter. There is no ruling that says one holdout per journey; three granularities are in play (journey, canvas, wave). |
| 5 · Unsubscribe and opt-out as the standing guardrail | Every shipped test reads unsubscribe, push opt-out and complaint rate per variant. A variant that wins on its primary KPI but spikes opt-outs fails. | PRACTICE, NOT RULED · now carried in all ten packs. Journey 5 carried none, and it sends to the never-transacted pool: unsubscribe rate and opt-out rate were added to its Test and Learn guardrails on 27 Jul 2026 (Sheran), baselines TBC: Analytics. Still TBC, Sheran: ratify it as a portfolio-level guardrail, and settle whether the standing term is unsubscribe or opt-out. |
Bar = build → live window; ▸ = results read. Colour = V2MOM objective. Snapshot of the hub roadmap (live version linked above).
Holdout-proven reads with a scheduled date. Standing cadence on every shipped test: +1wk · +2wk · +4wk (holdout-proven) · +3mo (scale / iterate / kill).
| Read date | Test | Sample · significance |
|---|---|---|
| 2026-09-14 | Early lapse-save nudge vs holdout (61 to 90d) | n=430 · 95% |
| 2026-09-21 | Value-tiered win-back vs holdout (lapsed 91 to 240d) | - |
| 2026-09-28 | Post-win 30-day programme vs holdout | - · 95% |
| 2026-10-06 | Churn-risk-timed lapsing-prevention vs holdout | n=550 · 95% |
One per journey release, per the test-discipline rule above (rule 2 in that table). Everything else in a journey is a phase-2 row gated on this read. Each journey's review pack is the current source for its own design; open it from the 10-journey list on Strategy on a page.
| Journey | The headline test | State |
|---|---|---|
| J1 First win to 2nd win | The win-moment one-job rework (receipt plus in-app: realised-value-led, referral stripped, single next-best action) vs the current 5-CTA send. | Guardrail floor is hard: open/click must not fall below 71.3% / 12.2% (confirmed, Braze/Lisa). Blocked until the 2nd-transaction KPI is standardised (TBC, Analytics/Ivi). |
| J2 Pre-travel & travel day | GRW-16 reassurance reframe A/B. Defensive: the day-before email is the only proven-incremental send, so the reframe must clear a non-inferiority floor. Current creative stays as control. Replace only on a win or a tie. | Headline pick still open (DL-046, Sheran picks, Ivi designs). Rollback floor TBC, Ivi + Lisa (DL-054). Three cells can hit one trip, so Analytics owns one interaction design. |
| J3 Funnel abandon | ACT-02 bid-add reassurance vs holdout; first-bid rate among never-bid dwellers. | Blocked on the dwell event and sizing. Baseline, MDE, significance and runtime all TBC, Analytics. |
| J4 Loss recovery | The D+7 gate itself: gated recovery vs a powered no-send control among 7-day non-returners. | DL-074. Guardrail: overall bid volume must not drop (the prior blanket read was −11.8%, confirmed). Ship-blocked on the pinned return definition (DL-069), which moves holdout power about 2.5x. |
| J5 Never-transacted, first win | The state-aware onramp as one release (upgrade-now copy plus the practice tile) vs holdout. Read conversion, never sessions. | Precondition: declare a winner on the existing education message or fold its control into this holdout. No unsubscribe or opt-out guardrail is set: see flag 5 above. |
| J6 Post-trip habit & advocacy | A journey-level holdout on the whole post-trip spine; next transaction within 60 days of a positive completed trip. | Baseline TBC, Analytics (Ivi). The broken survey link fix is a P0 before-and-after read, not a test cell. |
| J7 Lapsed win-back | The double read. Propensity-keyed waves vs a batch arm, each against a holdout within its own propensity band. | DL-103, adopted by the Journey 7 council 20 Jul. Band cut-offs and the decision rule fixed before launch. Cannot be sized until the churn-propensity model lands (launch-blocking, Analytics). |
| J8 First slip, early lapse-save | Frog voice vs brand voice, carried through the whole episode and read at episode level, plus a no-acknowledgement cell if volumes allow. | Cells TBC on volumes. A behavioural win that spikes opt-outs fails: the voice read gets a qual twin (sentiment and support themes per variant). |
| J9 Seasonal reactivation waves | Wave 1 vs its no-contact holdout. This is the funding read: if wave 1 cannot beat organic plus costs, the calendar does not scale. | Measured against an organic reactivation baseline of 0.84% (confirmed, Jul 2026). One baseline and pool definition must be published before the holdout is sized. |
| J10 Home re-entry | Recognition on vs off: state-resolved recognition card and tiles feed vs the current generic Home. Read transaction rate, never sessions. | DL-117, DESIGN stage. Needs a shared experiment calendar with Product so live product tests do not contaminate it. |
| Moment | Metric | Baseline | Band | Measurable? |
|---|---|---|---|---|
| GRW-16 | anxiety-contact rate / CSAT + conversion | 76.5% pre-travel neg; +16.9% incremental (confirmed) | PROVEN | Measurable: proven |
| ACT-13 | push opt-in capture rate among not-opted in-lead bidders | 32% not opted (~79k/yr); M29 rate TBC | CAPTURE | Blocked: needs opt-in→Tableau (P0) |
| GRW-01 | 2nd-transaction rate | 5% → 7.5% target | PLAUSIBLE | Needs holdout + correlation token |
| ACT-24 | opt-in capture + first-txn comprehension | clarity 54.7% 'somewhat clear'; 56% not opted at win | CAPTURE | Blocked, RFM segment + correlation token |
| ACT-07 | re-bid / recovery ≤30d | 159k losers/yr; current read −ve | PLAUSIBLE | Needs holdout (mandatory) |
| ACT-19 | new → first-transaction rate | 23%; Beginner's Luck 68.6% (confirmed) | PLAUSIBLE | Needs the never-bid walk + segments |
| GRW-17 | anxiety-contact / missed-coach tickets | 4,776 on-train value-failure tickets | PLAUSIBLE | Needs holdout + trip-state surface |
Full test design for the locked P1 moments. MDE, significance and runtime are set up front by Analytics, never post-hoc (post-hoc gave Q1 a 0-of-7 significance rate).