A frozen before-and-after scenario is a measurement design where you capture a baseline, hold it fixed, and compare a later state against it to attribute an outcome to something you changed. The word "frozen" matters because the baseline stays locked and, in the stronger versions, stays outside the influence of whatever system you are testing. That single choice separates a trustworthy read from a misleading one. A naive before/after on the same group is the weakest form because time itself can move your metric. A frozen counterfactual, built from a held-out region or a weighted blend of comparison markets, lets you subtract out the effect of time and isolate the treatment. For teams at 10xSearch measuring changes in AI search visibility or ad spend, the design choice determines whether a result is real signal or noise dressed up as a win. Below, Rick Janson walks through where each design fits and how to avoid corrupting the model you are trying to improve.
What Is A Frozen Before-and-After Scenario, Exactly?
A frozen before-and-after scenario is a measurement structure with two parts: a baseline captured before an intervention and held constant, and a post-intervention state compared against it to estimate the treatment's effect. The value of the design depends entirely on how well that frozen baseline represents what would have happened without the change.
There are several design buyers that share this structure. The simplest is the one-group pretest, posttest, where you measure the same units before and after a treatment. Stronger designs build a frozen counterfactual instead of relying on the same units over time. Difference-in-Differences (DiD) compares the before/after change in a treated group against the before/after change in an untreated control. Synthetic control constructs a weighted blend of comparison markets. Geo holdouts pause activity in withheld regions to preserve a clean baseline. A frozen-baseline retrain diagnostic holds model parameters fixed to separate data drift from model drift.
All of them rest on one attribution claim: the difference you observe is caused by the treatment and not by something else. The rest of this guide is about defending that claim.
Why Can A One-Group Before/After Comparison Mislead You?
A one-group before/after comparison can mislead you because nothing in the design rules out the possibility that the outcome would have changed anyway. When you measure the same units before and after a treatment, you are assuming that nothing else moved during that window. That assumption is rarely safe.
Three specific threats undermine it. The first is maturation, the natural drift of a metric over time regardless of what you did. The second is history, meaning concurrent events unrelated to your treatment. The longer the time lapse between the before and after observations, the more likely history becomes a threat, because more unrelated events can occur in that gap (UW-Madison Research Guides). The third is testing, where the act of measuring at the start conditions how the same units respond later.
A pretest is an improvement over measuring only after the fact, but it does not eliminate these threats. A before-after comparison of the same units is not automatically a causal comparison, because the change in time could account for the whole difference. For a quick, low-stakes internal read this is often acceptable. For a decision that reallocates budget, it is not.
How Does A Frozen Counterfactual Isolate The Treatment Effect?
A frozen counterfactual isolates the treatment effect by giving you a comparison group that experiences the same passage of time but not the treatment, so you can subtract out everything time did on its own. This is the core idea behind Difference-in-Differences.
DiD works in two steps. First you compute the before/after difference for the treated group. That single difference, however, captures both the treatment and any time-related movement mixed together. So you compute the same before/after difference for a control group that got no treatment, and you take the difference between those two differences. As a difference-in-difference study on AI-generated image detection describes, this second-order difference removes the confounding effect of time and leaves the treatment effect. Written out, the estimator is: (treated after − treated before) − (control after − control before).
The frozen part is the counterfactual. You lock in the pre-period behavior of the control before the treatment begins, then trust it to represent what the treated group would have done without intervention. That is what makes the comparison exogenous rather than self-referential. When you measure changes to how a site surfaces in AI answers, freezing an untouched comparison set is what turns a raw movement into an attributable result. Our work on before-and-after visibility measurement leans on exactly this structure.
Which Design Fits: Geo Holdouts, Synthetic Control, Or DiD?
The right design depends on whether you already run activity you can pause, whether a single clean control exists, and how much a wrong answer would cost. Each family solves a different version of the same problem.
Use a geo holdout when you already run spend and want the true incremental effect of turning a channel on or off. Geography is a clean boundary: you split at the regional level, run activity normally in control regions, and pause it in holdout regions. Because a person in one region does not see the ads paused in another, there is no cross-contamination and no audience overlap.
Use a synthetic control when no single clean control market exists. A synthetic control is a weighted blend of several markets that together track the treatment market's pre-period behavior. Instead of picking one imperfect match, you construct the best possible counterfactual from many markets combined. The classic example: by 2000, annual per-capita cigarette sales in California were estimated about 26 packs lower than they would have been absent Proposition 99, per the synthetic control method's original study.
Use DiD when you have a credible treated group and control group and the parallel-trends assumption holds. Geo experiments often estimate effects with Time-Based Regression, which predicts the counterfactual time series of the treated group. Note that TBR belongs to the synthetic control family and, as the Trimmed Match geo experiment paper explains, still requires some untestable assumptions even with randomized geo assignment.
| Design | Best-fit use | Key requirement |
|---|---|---|
| One-group pretest, posttest | Quick, low-stakes internal read | Assume nothing else changed |
| DiD / geo holdout | Attributing lift for budget or optimizer decisions | Parallel pre-period trends |
| Synthetic control | No single clean control market exists | Good pre-period fit from blended markets |
The reading of the result is as important as the design. An iROAS of 2, as defined in Google Ads Help, means every $1 invested generated $2 of net new conversion value that otherwise would not have existed. That is incremental value, not total value, which is the whole point of freezing a counterfactual.
How Do You Keep A Measurement From Poisoning Your Learning Loop?
You keep a measurement from poisoning your learning loop by holding out a slice of activity from the model's influence, so the outcome you feed back as training signal is not just an echo of what the model already decided. This is the failure mode most teams miss.
Here is the mechanism. When a model chooses the treatment and the resulting before/after outcome is fed back as training data, the baseline is no longer independent of the model. As research on feedback loops in recommender systems shows, feedback loops induce a sampling bias that makes the collected dataset a missing-not-at-random problem across multiple time steps. In plain terms, the model only sees outcomes for the choices it already favored, so its next round of training reinforces its own prior behavior.
That reinforcement compounds. Work on presentation bias in multimodal learning documents that new labels depend on the model's earlier behavior, so they may not reflect true preferences, and the bias amplifies as more training loops complete on biased data. Each cycle looks like improvement while the system drifts further from reality.
The fix is a frozen, exogenous counterfactual. Reserve a held-out region or set of units that the model cannot touch, and measure against it. Because that slice stays outside the deciding system, its outcomes remain a clean signal rather than a self-fulfilling one. When we run prompt testing for AI search and visibility reporting, the held-out set is what keeps the measurement honest across cycles.
One more safeguard applies to any surprising result: treat breakthrough wins with extra skepticism. Twyman's Law holds that any figure that looks interesting or different is usually wrong, so a result that looks too good should trigger validity checks before it drives a decision. An A/A validation test, where you compare two untreated groups that should show no difference, is the cheapest way to catch a broken measurement platform before you trust it.
Frequently Asked Questions
What is a frozen before-and-after scenario?
A frozen before-and-after scenario is a measurement design where a baseline is captured and held fixed, then compared against a later state to attribute an outcome to a treatment. In the stronger versions the baseline is also kept outside the influence of the system being tested, which prevents the result from becoming self-referential.
Why is a simple before/after comparison on the same units considered weak?
A simple before/after on the same units is weak because it cannot separate the treatment from the passage of time. Maturation, concurrent history events, and testing effects could each produce the observed difference on their own, so the design does not eliminate the possibility that the outcome would have happened without any treatment at all.
What is the parallel-trends assumption in difference-in-differences?
The parallel-trends assumption is the requirement that, in the absence of treatment, the difference between the treatment group and the control group stays invariant over time. The two groups' metrics need not be equal, but their movement should track together. If pre-period trends diverge, freezing that control as a counterfactual produces a biased estimate.
When should I use a geo holdout versus a synthetic control?
Use a geo holdout when you already run spend and can pause it in withheld regions, since geography gives a clean boundary with no audience overlap. Use a synthetic control when no single clean comparison market exists, so you build a weighted blend of several markets that together track your treatment market's pre-period behavior.
How does a feedback loop bias a before-and-after measurement?
A feedback loop biases a measurement when the model that chose the treatment also learns from the outcome, making the baseline non-independent. This creates a missing-not-at-random problem, because the system only observes results for choices it already favored. The bias amplifies with each training cycle, so the loop reinforces its own prior behavior rather than reality.
Why should surprising positive results get extra scrutiny?
Surprising positive results deserve extra scrutiny because of Twyman's Law: any figure that looks unusually interesting or different is usually wrong. A breakthrough number is more likely to reflect a measurement bug, a broken assumption, or a contaminated baseline than a genuine effect, so run validity tests such as an A/A check before acting on it.