A Normal-Variation Worksheet for AI Visibility Rates

Several guides on this site end with the same instruction: compare a change against your normal variation. This is the worksheet that computes it. It needs nothing but the rows you're already recording and a spreadsheet.

The worksheet, filled with illustrative data

Four quiet periods, one prompt set of 25 prompts, three runs per prompt per assistant, so 75 answers per assistant per period. Each cell is mentions out of answers. Every number is synthetic.

The calculation

Per assistant, over quiet periods k:
  rate_k      = mentions_k / answers_k
  quiet_range = min(rate_k) .. max(rate_k)
  pooled_mean = sum(mentions_k) / sum(answers_k)
  se          = sqrt(pooled_mean * (1 - pooled_mean) / answers_per_period)

New period:
  flag   if rate_new is outside quiet_range
         AND |rate_new - pooled_mean| > about 2 x se
  confirm only if it holds in the next period
         and the control set moved by less

Reading a new period

Say period 5 follows a batch of page changes. Assistant A comes in at 20/75, about 27%. That's roughly nine points above its pooled mean and five above its highest quiet period; its standard error at 17% on 75 answers is about 4.4 points, so the move is a little over two of them. Worth investigating. Assistant C comes in at 27/75, 36%. That's six points above its mean, but only one above its own highest quiet period, and C's standard error at 30% is about 5.3 points. That's barely outside C's quiet range and only about one standard error from its mean: not enough to flag. Similar-looking moves, different verdicts, and the difference is entirely in each assistant's own history.

Why the formula understates the noise

The binomial standard error assumes independent draws. Repeated runs of the same prompt in the same window probably aren't fully independent, since they share a model version and an index state, and prompts differ from one another. The formula also knows nothing about platform changes. So treat it as a floor on the noise, and trust the observed quiet-period range more than the formula whenever the two disagree.

What counts as a quiet period

One with no shipped changes to the pages your prompts map to, no change to the prompt set or the sampling surfaces, and no platform change you know of. Perfectly quiet periods are rare. Use the quietest you have, label the worksheet with what did happen in each, and accept that a noisy history produces a wide range, which is itself an honest finding.

FAQ

Can I compute this per prompt? — Only with more runs than most teams have. With three runs a period, a single prompt's rate can only be 0%, 33%, 67% or 100%, so per-prompt variation is mostly arithmetic. Work per assistant first.

What if we've already shipped changes everywhere? — Then this prompt set has no quiet periods, and normal variation is unknown. Say that, and set up a control set before the next round.

Does a bigger sample fix it? — It narrows sampling noise, which falls with the square root of the number of runs: four times the runs halves the standard error. It doesn't touch platform change, which no sample size removes.