A change that shipped is not a change that worked. Between those two statements sit a window, a comparison and a decision.
What a fair comparison needs
- A baseline, recorded before the change.
- A comparison set, so the whole site does not move at once.
- A window, agreed before the change ships.
- A source that can read the metric you predicted.
Without the fourth, no verdict is possible. That is a fact about measurement, not a shortage of effort.
The three verdicts worth using
- Real effect. The metric moved beyond the comparison’s own variation.
- Inconclusive. The window closed without enough signal either way.
- Not measured. No source was connected, or the window never closed.
A worked change
Cedar and Slate Interiors, a fictional client. The change rewrote three service titles. The prediction was clicks, up, over 28 days.
change: rewrite three service page titles
pages: /services/kitchens, /services/bathrooms, /services/joinery
shipped: 2026-08-04
window: 2026-08-04 to 2026-08-31, 28 days
source: search console, cedarandslate.example
prediction: clicks up over 28 days, direction only, no number
baseline, 2026-07-07 to 2026-08-03, 28 days
impressions 2,940
clicks 88
click rate 2.99%
avg position 16.3
result, 2026-08-04 to 2026-08-31, 28 days
impressions 3,210
clicks 91
click rate 2.83%
avg position 15.8
comparison set, pages not changed
impressions +4.1%
clicks +2.2%
verdict: inconclusive
reason: the change moved 3.4% and the unchanged pages moved 2.2%.
The gap is smaller than the week-to-week swing on this site.
notes: one competitor ran a promotion in the second half of August.
Recorded as a confounder, not as a result.
Most changes on a small site land here. A tool that cannot say this did nothing will tell you everything worked.
Why the confounder matters
Anything else that moved during the window can explain the numbers. Writing it down keeps the next reader from treating the change as proven.
Acceptance: the record is complete when
- the window was agreed before the change shipped
- the baseline is a real reading, with dates
- the comparison set is named, and its movement is shown
- confounders are written down
- the verdict is one of the three, never a story
- the next decision is recorded, even when the verdict is inconclusive
Troubleshooting
| Symptom | Likely cause | What to do |
|---|---|---|
| The metric moved a lot | The site changed in other ways too | Read the comparison set before claiming anything |
| No result at all | The window is too short for the volume | Extend the window, or accept inconclusive |
| The prediction has no source | Nothing connected measures it | Fix that before shipping the next change |
| The client wants a number | The site has too little volume | Report the direction, with the error bars named |
| Every change looks successful | Nobody checked the comparison set | Add the comparison set to the record |
Limits
A single site rarely has enough volume for a clean answer in four weeks.
That argues for a longer window and a smaller change. It is not a reason to read the result early and call it a win.