Insights · Marketing finance

Incrementality testing: your media grades itself

The cookie deadline was cancelled twice. Consent held at roughly two thirds. And measurement still got worse, because nothing disappeared: the signal moved inside the companies that sell you the media, and they are the ones who now produce the number.

Incrementality testing is the practice of withholding a channel from a randomly chosen share of the market and then measuring the difference in sales between the exposed and the withheld group. It is the only method that produces a number an advertiser owns rather than receives. It matters more in 2026 than it did in 2020, and not for the reason the industry spent five years preparing for.

The deadline was cancelled twice

On 22 April 2025, Anthony Chavez, VP Privacy Sandbox, wrote that Google had “made the decision to maintain our current approach to offering users third-party cookie choice in Chrome, and will not be rolling out a new standalone prompt for third-party cookies.” Six months later, on 17 October 2025, the same author retired ten technologies: the Attribution Reporting API, IP Protection, On-Device Personalization, Private Aggregation, Protected Audience, Protected App Signals, Related Website Sets, SelectURL, SDK Runtime and Topics. Four survived: CHIPS, FedCM, Private State Tokens and an interoperable attribution standard. The stated reason was their expected value “and in light of their low levels of adoption.”

So the five-year programme to replace the third-party cookie ended, and the third-party cookie did not. If your 2027 media plan still contains a workstream called cookie readiness, it is preparing for an event that was formally called off twice.

What the cookie was actually worth

That is not the same as saying cookie loss costs nothing. The UK Competition and Markets Authority published a summary of the testing on 13 June 2025, and it is the closest thing the industry has to an audited answer.

Publisher revenue per impression fell 27 percent on traffic scoped to Google Ad Manager and 15 percent on AdSense, with the Privacy Sandbox tools recovering 13 and 3 percentage points respectively. Advertiser spend fell 14 percent on DV360 and 11 percent on Google Ads, with about 10 points recovered in each. Third-party testers found far worse: advertiser spend down 42, 60 and 67 percent, impressions down 35, 47, 67 and 69 percent, and latency up by as much as 200 percent. The CMA summary of the sell-side is the sentence to keep: publisher revenue per impression was around 30 percent lower without third-party cookies even accounting for the availability of the Privacy Sandbox.

Read what those numbers are, though. They are prices. A publisher earns less per impression because the buyer can no longer tell who is on the other side, and the market prices the absence of information correctly. That is inventory being repriced. It is not your attribution report getting worse. Your attribution report was never good, and the cookie was never what made it good.

A quarter of Europe was never in the cookie

In September 2026, StatCounter put Chrome at 60.01 percent of European browsing, Safari at 19.94, Edge at 7.29 and Firefox at 4.3. Safari has blocked third-party cookies by default since 2020 and Firefox partitions them, so in neither does one work for cross-site identity. That is 24.2 percent of Europe, and 18.5 percent of Poland, where the post-cookie world arrived without a press release and nobody renamed a dashboard.

Then multiply by consent. The 2026 Didomi benchmark, built on 2025 data, puts opt-in at 55.7 percent in Western Europe and 67.6 percent in Eastern Europe, with between 21.7 and 27.4 percent of users simply leaving the banner unanswered. Three quarters of European traffic in a browser that permits the cookie, six in ten of those consenting: the observable share of your audience has been somewhere near half for a long time. No dashboard shows that multiplier, because a dashboard reports a total, not a coverage rate.

Advertising returns are statistically invisible

Here is the part that should end most attribution arguments. Randall Lewis and Justin Rao published a paper in the Quarterly Journal of Economics with an unusually honest title: On the Near Impossibility of Measuring the Returns to Advertising. Working on real campaigns, they found that the sample size required for an experiment to produce usefully narrow confidence intervals is typically in excess of ten million person-weeks. The ratio of advertising impact to the standard deviation of sales was 0.0047, giving an R-squared of 0.0000054. To separate a modestly profitable campaign at 10 percent return from breakeven, the median retail experiment they studied would have to be 61 times larger. For financial services, 1241 times larger.

Sit with the implication. If a randomised experiment at that scale is the minimum needed to see the effect, a model fitted to observational clickstream data cannot see it at all. The authors make the point directly: a very small amount of endogeneity would severely bias estimates of advertising effectiveness, because the selection bias required to explain even a trivial share of the variance is an order of magnitude larger than the true treatment effect. The noise is not a tracking problem that better identity resolution fixes. The noise is the sales distribution itself.

The number on your dashboard did not come from your data. It came from the company that sold you the media.

Which is why the number comes from the seller

George Akerlof described the structure in 1970. In a market where the seller knows the quality of the good and the buyer does not, the prediction is specific: buyers pay the average, genuine quality earns less than it is worth, and genuine quality eventually leaves the market.

Media measurement fits the premise exactly. The platform holds the counterfactual, because the platform decided which users would see the ad and therefore knows which of them were going to convert anyway. The advertiser holds a report. And the report is produced by the party being paid.

Nobody has to lie for this to go wrong. Last click, seven-day click with one-day view, modelled conversions: each is defensible on its own terms. Put four channels on their own windows and the attributed revenue sums to more than the revenue. Everyone is honest and the total is false.

The prediction, translated into a media plan: the channels that are easiest to attribute report the best returns and get funded. Brand search, retargeting, on-site retail media. The channels that genuinely create demand report worst and get cut, because broad reach works on a lag and across a population the tag never sees. We have made this argument on retail media, where eight times reported commonly lands between one and two times incremental, and the growth consequence is the one we set out in penetration, not loyalty. Over two planning cycles the budget migrates from media that works to media that counts. That is Akerlof's exit of good quality, with a media plan attached.

eBay turned it off and nothing happened

The cleanest demonstration is twelve years old and still unanswered. Blake, Nosko and Tadelis ran a field experiment on eBay's United States paid search, then running at 51 million dollars a year, switching it off across 68 markets against matched controls for 60 days. It was published in Econometrica in 2015.

On brand keywords, 99.5 percent of the forgone paid click traffic was immediately captured by natural search. The ads were buying traffic eBay already had. Across the whole paid search regime the effect on sales was 0.66 percent, with a 95 percent confidence interval from minus 0.42 to 1.74 percent, implying a return on investment of minus 63 percent with an interval from minus 124 to minus 3 percent. The paper rejects the hypothesis that the channel produced any positive short-run return at all.

The finding inside the finding is the one worth repeating in a planning meeting. The effect was largest for users who had never bought on eBay, rose again for those absent more than a year, and sat near zero for people who bought regularly. The money was going to the customers who needed no persuasion, and the measurement called that success, because those customers converted.

Incrementality testing, costed

The objection to holdouts is always the same: we cannot afford to switch media off. Fine. Cost it.

Geo holdout, worked	Value
Annual digital spend	10,000,000
Share of markets suppressed	10%
Duration	6 weeks
Spend withheld	115,000
True incremental return assumed	2.0x
Contribution margin	30%
Contribution forgone	69,000
Cost of the experiment, share of annual budget	0.69%

Currency is irrelevant, every line is a ratio. Contribution rather than revenue, because revenue is not the part you keep - the argument we set out in the discount nobody costed.

Seven tenths of one percent. And that is the worst case, the case where the suppressed media genuinely works at two times incremental. If the channel is the eBay brand-search case, the holdout costs nothing at all, because it hands the spend back. eBay's experiment did not cost eBay money. It found a channel running at minus 63 percent.

So the choice is not between spending on measurement and spending on media. It is between seven tenths of a percent and an error that, in the published experiments, has run to the entire return of a channel.

How we count it

This sits across two of our lenses. Digit It rebuilds the measurement stack so that every number on the dashboard carries three named things: who produced it, what the counterfactual was, and what the coverage rate is. Most stacks fail the first question, and the first question is the cheapest to answer. Cashstream, our portfolio P&L lens, puts the surviving numbers where they belong, against contribution rather than revenue, and prices the experiment against the decision it informs. That is how a test worth 0.7 percent of budget gets approved in a room that has just refused a 3 percent budget increase.

Five things to do, in order. Write down the coverage rate: browser mix times consent rate. If it lands near half, every channel report you read is at least half a model. Sum the attributed revenue across all channels and divide by actual revenue; the gap is the overclaim, and you can produce it this week with no new tooling. Pick the channel with the most suspiciously consistent return, because consistency is the tell - real media is volatile, counting is smooth. Hold it out: geo split, six weeks, 10 percent, and agree the read before you start, since the read is where incrementality tests usually die. Then ask who in the organisation owns the answer. If the party that buys the media also reports the result, you have Akerlof's problem, and no amount of dashboard work will fix it.