SearchPilot Claims a Controlled Lift in AI Impressions

You have probably shown a client a chart where AI visibility climbs after a site change. The line went up and the change got the credit. The trouble is that the line was likely going up anyway.

SearchPilot says it has a way around that. In an X post on October 21, 2026, Will Critchlow of SearchPilot said the company is now using Google's Search Console AI data inside its controlled split tests. Critchlow wrote that SearchPilot is "detecting AI visibility impacts with statistical confidence for the first time."

I think holding some pages back as an unchanged control is the right method. What the tests measure is still impressions, and being shown is not the same as being picked.

Critchlow posted one chart and a short explanation. That is what he has shared so far.

His chart shows an AI visibility line rising both before and after a website change. On its own, that proves nothing. He made the point himself in the post. Critchlow wrote that one problem with most AI visibility tracking is that there is "no way to tell whether the numbers are changing because of things you did, or just because of <gestures wildly>." In a reply, Critchlow added that "There is also such strong macro growth in AI that if you just look before after any change, you're probably up!"

The fix, as SearchPilot describes it, is a control group. A set of pages stays unchanged, and their trend produces a forecast of what the changed pages would have done anyway. SearchPilot reads the gap between that forecast and the real line as the effect, with a credible interval around it. Critchlow says the change they made helped.

What the data can and cannot show

The raw material is Google's own. Google launched generative AI performance reports in Search Console on June 3, 2026 and, per its help page, rolled them out to all sites on August 31. They count impressions in AI Overviews and AI Mode on Google Search and break them down by page, country, device and date. They leave out Search Labs experiments, and they say nothing about ChatGPT, Perplexity or any other engine. Asked whether the chart shows Search Console impressions, Critchlow replied yes.

Page-level data is what makes a split test possible, because pages can be assigned to buckets and tracked one by one. The same reports have limits. As Shortlist's explainer puts it, the report has no click column, no click-through rate and no queries. Growth marketer Cody Schneider made the same point on October 19. "one catch. it's impressions, not clicks."

So a test like this can show that a change caused more impressions. It cannot show that the page was the one the AI recommended, or that anyone visited. Critchlow replied that SearchPilot has been running click tests too.

Two more objections

The first is spillover. The German firm OIT GmbH wrote that "AI answers often cite a different URL from the same site than the one you changed, so a variant page can gain visibility partly by pulling it away from a control page." If that happens, the test page looks better and the control looks worse, and the effect is overstated.

Critchlow answered in the thread. "It's early days for this analysis, but in general, while that effect may lead to overstating the impact, I would expect it not to change the direction."

That is an expectation, not a result. The second objection is duration. One r/TechSEO poster reported a spike in generative AI impressions that arrived with almost no extra clicks, and a lift that fades is not much of a win. Critchlow has not said how long the lift in his chart lasted.

The test itself is unpublished. Critchlow said he cannot remember which test he redacted and anonymised for the post, and that full write-ups wait on client permission. The site, the change and the size of the lift are all unknown. SearchPilot runs these tests for its clients and has a stake in them being believed. Writer Kaleigh Moore called it "one of the most interesting experiments on AI visibility I've seen because a control group is always missing from the data set, and necessary to give performance context." That is praise for the design, not a check of the result.

The idea travels without SearchPilot

Export page-level AI impressions from Search Console, split similar templated pages into two buckets, and watch how far apart the buckets drift when you change nothing. That drift is a rough noise floor. Then change one bucket, leave the other alone, and compare the trends. A gap that clears the noise is worth chasing, though it would still not prove the change caused it.

Disclosure: my agency, ViewEngine, sells AI visibility work and builds its own tracking tools.

My read is that the field has to go this way. AI answers vary from run to run, which makes one before and after chart nearly worthless. A control group is the first honest way to separate your work from the tide. Until someone shows that more AI impressions lead to more picks and more branded searches, I will treat results like this as a sign you moved a number Google counts. Whether it moved a buyer is still an open question.