Fix the method before the first reading
A number you cannot repeat is a story. Write the method down, and repeat it on the same day of the week.
The procedure
1. choose 20 to 50 prompts, in the words a buyer would use
2. include branded and plain questions, and count them separately
3. freeze the set, and version it when it changes
4. run every prompt in one sitting, on one day
5. record the answer text beside each prompt
6. mark each answer: named, cited, or absent
7. keep the raw answers, so a later reading can be compared
Two readings, filled in
Brightwater Dental, twelve prompts, version 1.
reading A, 2026-09-14
prompts 12, answers 12
named 5, cited pages /emergency /implants /childrens
absent prompts 3, 4, 5, 7, 9, 11, 12
reading B, 2026-09-28
prompts 12, answers 12
named 7, cited pages /emergency /implants /childrens /pricing
absent prompts 4, 5, 7, 9, 12
comparison
same set, same count, fourteen days apart
difference: two answers, both cost questions
page added to the cited set: /pricing
what changed on the site in between
one paragraph about implant cost was added to /pricing on 2026-09-17
what that does not prove
two answers is inside the swing of a small sample, and the model
version was not recorded. Reported as a reason to look, not a result.
How to compare readings honestly
- Compare the same prompt set, or say the set changed.
- Compare the same weekday and a similar time of day.
- Report the count, not only the percentage. Small samples move a lot.
- Name anything else that changed: a new page, a new offer, a model update.
Acceptance: the method is repeatable when
- the prompt set is stored with a version and a date
- the answers are kept, not only the counts
- the run conditions are recorded
- branded and unbranded counts are separate
- the report names what a change cannot prove
Troubleshooting
| Symptom | Likely cause | What to do |
|---|---|---|
| The prompt set keeps changing | Nobody froze it | Version it, and re-run from that version |
| One prompt dominates the count | The set is unbalanced | Cap each topic at two prompts |
| Answers are not reproducible at all | Personalisation, or a model change | Record both, and widen the error bars |
| A client reads the delta as a result | The window is too small | Show the raw answers side by side |
| Nobody acts on the reading | No decision was attached | Name one page to read, not all of them |
Limits
This procedure measures answers you can collect. It does not measure what a customer did after reading one.
Two readings are a comparison. Three are the beginning of a direction.