Why Are Marketers Turning To Quasi Geo-Lift Experiments? (And How to Plan Them)

Note: Geo-Lift R code at the end of the article.

of my career, I have used quasi-experimental designs with synthetic control groups to measure the impact of business changes. At Trustpilot, we switched the position of a banner on the homepage and retrospectively saw signals pointing to a decrease in our main demand metric. By modeling how performance would have looked without the change and comparing it to what happened, we got a clear read of incremental decline. At Zensai, we decided to treat some of our web forms and confirmed in a similar way whether the change had a positive impact on user-reported ease of completion.

This approach works wherever you can split something into treatment and control groups: individuals, smartphones, operating systems, forms, or landing pages. You treat some. The others help you model what would have happened without treatment. Then you compare.

When you cannot randomize at the individual level, geography makes a great unit of analysis. Cities and regions are straightforward to target via paid social and their borders help contain spillovers.

There are three practical designs for isolating incremental impact:

1. Randomized controlled trials (RCT) like conversion or brand lift tests are appealing when you can use them as they run inside big ad platforms. They automate random user assignment and can deliver statistically robust answers within the platform’s environment. However, unless you leverage conversion API, their scope is constrained to the platform’s own metrics and mechanics. More recently, proofs (1, 2, 3) have emerged that algorithmic features may compromise the randomness of assignment. This creates “divergent delivery”: ad A gets shown to one type of audience (e.g., more men, or people with certain interests), while ad B gets shown to another type. Any difference in CTRs or conversions can’t be attributed purely to the ad creative. It’s confounded with how the algorithm delivered the ads. On Google Ads, Conversion Lift typically requires enablement by a Google account representative. On Meta, eligible advertisers can run self-serve Conversion Lift tests in Ads Manager (subject to spend and conversion prerequisites), and Meta’s 2025 Incremental Attribution feature further lowers friction for lift-style measurement.
2. Geo-RCTs (Randomized Geo Experiments) use geographies as the unit of analysis with random assignment to treatment and control. There is no need for individual-level tracking, but you need a sufficiently large number of geos to achieve statistical power and confident results. Because assignment is randomized, you don’t build a synthetic control as you do in the next type of experiment.
3. The Quasi Geo-Lift Experiment approaches the same question using geos as the units of analysis, with no need for individual-level tracking. Unlike Geo-RCTs that require randomization and more geos, this quasi-experimental approach offers three key advantages with no required randomization needed: it a) works well with fewer geographies (no need for many geos as with Geo-RCTs), b) enables strategic market selection (direct control over where and when treatment is applied based on business priorities), and c) accommodates retrospective analysis and staggered rollouts (if your campaign already launched or must roll out in waves for operational reasons, you can still measure incrementality after the fact since randomization isn’t required). The synthetic control is constructed to match pre-intervention trends and characteristics of the treatment unit, so any differences after treatment begins can be attributed to the campaign’s incrementality. But: successful execution requires strong alignment between analytics and performance marketing teams to ensure proper implementation and interpretation.

The benefits of the *Quasi Geo-Lift Experiment* are great. So to help you understand when you might use a quasi-experiment in your marketing science function, let’s consider the following example below.

Quasi Geo-Lift Experiment Example

Consider a ride-hailing company such as Bolt evaluating a new channel in Poland. The business has had a new channel on radar for some time and now wants to test it out. There is a need to understand how a potentially new campaign on this new channel called TweetX affects rides, and what minimum budget is required to detect the impact. The available dataset is a daily ride panel covering 13 Polish cities starting in 2022. The proposed start date is 2023–09–18, and activation can be national or city-based. Practical constraints are present: users cannot be enrolled into individual treatment and control groups, since the new channel campaign targets only new riders. This rules out user-level A/B testing. Additionally, privacy conditions prevent reliance on tracking-based attribution. Historical priors from other channels in Poland suggest a cost per ride between €6 and €12, CPMs between €2 and €4, and an average profit per ride of €6. A recent benchmark monthly budget for a comparable channel, Snapcap, was a maximum of €5,000/month.

These conditions favor a quasi-experimental geo design that measures business outcomes directly, without relying on user-level tracking or dependencies on platforms like Meta or Google, but instead using commonly available historical data such as rides, sales, conversions, or leads.

To perform the Geo-Lift test with a synthetic control group design on our example, we will use the GeoLift package in R, developed by Meta. The dataset I will be using is structured as daily ride counts across 13 cities in Poland. 

Best practices for your data when using a Quasi Geo-Lift Experiment.

• Use daily data instead of weekly when possible.
• Use the most detailed location data available (e.g., zip codes, cities).
• Have at least 4–5 times the test duration in stable, pre-campaign historical data (no major changes or disruptions – more on this later in the operationalization chapter below!)
• Have at least 25 pre-treatment periods with a minimum of 10, but ideally 20+ geo-units.
• Ideally, collect 52 weeks of history to capture seasonal patterns and other factors.
• The test should last at least one purchase cycle for the product.
• Run the study for at least 15 days (daily data) or 4–6 weeks (weekly data).
• Panel data (covariates) is helpful but not required.
• For every time/location, include date, location, and KPIs (no missing values). Extra covariates can be added if they also meet this rule.

Planning Your Quasi Geo-Lift Experiment

Designing a quasi geo-lift experiment is not just about running a test,  it’s about creating a system that can credibly link marketing actions to business outcomes. 

To do this in a structured way, the launch of any new channel or campaign should be approached in stages. Here is my ABCDE framework that can help you with it:

(A) ASSESS

• Establish how incrementality will be measured and which method will be used.

(B) BUDGET 

• Determine the minimum spend required to detect an effect that is both statistically credible and commercially meaningful.

(C) CONSTRUCT 

• Specify which cities will be treated, how controls will be formed, how long the campaign will run, and what operational guardrails are needed. 

(D) DELIVER

• Convert statistical results into metrics and report outcomes as a readout.

(E) EVALUATE

• Use results to inform broader decisions by updating MMM and MTA. Focus on calibrating, stress-testing, replicating, and localizing for rollout.

(A) ASSESS the marketing triangulation variable in question by drilling down into it.

1. Marketing triangulation as a starting point.

Start by evaluating which part of the marketing triangulation you will need to break down. In our case, that would be the incrementality piece. For other projects, drill down into MTA and MMM similarly. For instance, MTA uncovers heuristic out-of-the-box techniques like last click, first click, first or last touch decaying, (inverted) U-shape, or W-shape, but also data-driven approaches like Markov chain. MMM can be custom, 3rd party, Robyn, or Meridian, and involves additional steps like saturation, adstock, and budget reallocation simulations. Don’t forget to set up your measurement metrics.

2. Incrementality is operationalized through a geo-lift test. 

The geo-lift test is the practical expression of the framework because it reads the outcome in the same units the business manages, such as rides (or sales, conversions, leads) per city per day. It creates treatment and control just as a classic randomized study would, but it does so at the geographic level. 

This makes the design executable across platforms and independent of user-level tracking, while it allows you to study your preferred business metrics.

3. Recognition of two experimental families of the Geo-Lift test: RCTs and quasi-experiments. 

Where in-platform RCTs such as conversion lift tests or brand lift tests exist (Meta, Google), they remain the standard and can be leverage. When individual randomization is infeasible, the geo-lift test then proceeds as a quasi-experiment.

4. Identification relies on a synthetic control method. 

For each treated city, a weighted combination of control cities is learned to reproduce its pre-period trajectory. The divergence between the observed series and its synthetic counterpart during the test window is interpreted as the incremental effect. This estimator preserves scientific rigor while keeping execution feasible and auditable.

5. Calibration and validation are explicit steps, not afterthoughts. 

The experimental estimate of incrementality is used to validate that attribution signals point in the right direction and to calibrate MMM elasticities and adstock (via the calibration multiplier), so cross-channel budget reallocations are grounded in causal truth.

6. Measure the impact in business terms. 

In the planing phase, the core statistic is the Average Treatment Effect on Treated (ATT), expressed in outcome units per day (e.g., rides per city per day). That estimate is translated into Total Incremental Rides over the test window and then into Cost per Incremental Conversion (CPIC) by dividing spend by the total number of incremental rides. Minimum Detectable Effect (MDE) is reported to make the design’s sensitivity explicit and to separate actionable results from inconclusive ones. Finally, Net Profit is calculated by combining historical rider profit with the incremental outcomes and CPIC.

The total incremental leads can be multiplied by a blended historical conversion rate from lead to customer to estimate how many new customers the campaign is expected to generate. That figure is then multiplied by the average profit per customer in dollars. This way, even when revenue is realized downstream of the lead stage, the experiment still delivers a transparent estimate of incremental financial impact and a clear decision rule for whether to scale the channel. All other metrics like ATE, total incremental leads, cost per incremental lead, and MDE will be calcalted in a similar fashion.

(B) BUDGET for your quasi-experiment.

Budget estimation is not guesswork. It is a design choice that determines whether the experiment yields actionable results or inconclusive noise. The key concept is the Minimum Detectable Effect (MDE): the smallest lift the test can reliably detect given the variance in historical data, the number of treated and control cities, and the length of the test window.

In practice, variance is estimated from historical rides (or sales, conversions, or leads in other industries). The number of treated cities and test length then define sensitivity. For example, as you will see later, treating 3 cities for 21 days while holding 10 as controls provides enough power to detect lifts of about 4–5%. Detecting smaller but statistically significant effects would require more time, more markets, or more spend.

The Geo-Lift package models these power analyses and then prints the budget–effect–power simulation chart for any experiment ID.

The budget is aligned with unit economics. In the Bolt case, industry priors suggest a cost per ride of €6–€12 and a profit per ride of €6. Under these assumptions, the minimum spend to achieve an MDE of roughly 5% comes to €3,038 for 3 weeks, or €48.23 per treated city per day. This sits within the €5,000€ benchmark budget but, more importantly, makes explicit what effect size the test can and cannot detect.

Framing budget this way has two advantages. First, it ensures the experiment is designed to detect only effects that matter for the business. Second, it gives stakeholders clarity: if the result is null, it means the true effect was more likely smaller than the threshold, not that the test was poorly executed. Either way, the spend is not wasted. It buys causal knowledge that sharpens future allocation decisions.

(C) CONSTRUCT your quasi-experiment design.

Designing the experiment is about more than picking cities at random. It is about building a layout that preserves validity while staying practical for operations. The unit of analysis is the city-day, and the outcome is the business metric of interest, such as rides, sales, or leads. Treatment is applied to selected cities for a fixed test window, while the remaining cities serve as controls. 

Geo-Lift will model and group the ideal city candidates for your treatment.

Control groups are not just left as-is. 

They can be refined using a synthetic control method. Each treated city is paired with a weighted combination of control cities that reproduces its pre-test trajectory. When the pre-period fit is accurate, the post-launch divergence between observed and synthetic outcomes provides a credible estimate of incremental effect.

Operational guardrails are critical to protect signal quality. 

City boundaries should be fenced tightly in settings to reduce spillovers from commuters. Also consider strict exclusion of control cities from treatment cities in the settings and vice-versa. There should not be a specific targeting and lookalike audiences applied to the campaign. Local promotions that could confound the test are either frozen in treated geographies or mirrored in controls. 

Creatives, bids, and pacing are held constant during the window, and outcomes are only read after a short cooldown period due to the marketing adstock effect. Other business relevant factors should be considered, e.g. in our case, we should check the supply driver capacity in advance to ensure additional demand can be served without distorting prices or wait times.

Constructing the test means choosing the right balance.

The Geo-Lift package will do the heavy lifting for you by modeling and building all optimal experiments.

From your power analysis and market selection function code, choosing the right experiment setup is a balance between:

• number of cities
• duration
• spend
• desired incremental uplift
• profit
• stat. significance
• test vs control alignment 
• smallest detectable lift 
• other business context

A configuration of three treated cities over a 21-day period, as in the Bolt example, provides sufficient power to detect lifts of ~4–5%, while keeping the test window short enough to minimize contamination. Based on previous experiments, we know this level of power is adequate. In addition, we need to remain within our budget of €5,000 per month, and the expected investment of €3,038.16 for three weeks fits well within that constraint.

(D) DELIVER the post-experiment readout.

The final step is to deliver results that translate statistical lift into clear business impact for stakeholders. The experiment’s outputs should be framed in terms that both analysts and decision-makers can use.

At the core is the Average Treatment Effect on Treated (ATT), expressed in outcome units per day such as rides, sales, or leads. From this, the analysis calculates total incremental outcomes over the test window and derives Cost per Incremental Conversion (CPIC) by dividing spend by those outcomes. The Minimum Detectable Effect (MDE) is reported alongside results to make the test’s sensitivity transparent, separating actionable lifts from inconclusive noise. Finally, the analysis converts outcomes into Net Profit by combining incremental conversions with unit economics.

For lead-based businesses, the same logic applies but the net profit: for that, the total incremental leads can be multiplied by a blended conversion rate from lead to customer, then by average profit per customer, to approximate the net financial impact.

Be cautious when interpreting results to stakeholders. Statistical analysis with p-value provides evidence, not absolute proof, so phrasing matters. The GeoLift uses Synthetic Control/Augmented Synthetic Control with frequentist inference.

A common but misleading interpretation of statistical significance might sound like this:

“The result of the quasi Geo-Lift experiment proves that rides increased by 11.1% because of the campaign with a 95% probability.”

This interpretation is problematic for several reasons:

• It treats statistical significance as proof.
• It assumes the effect size is exact (11.1%), ignoring the uncertainty range around that estimate.
• It misinterprets confidence intervals.
• It leaves no room for alternative explorations that might create value for the business.
• It can mislead decision-makers, creating overconfidence and potentially leading to risky business choices.

When performing quasi Geo-Lift test, what a statistical test actually tests?

Every statistical test depends on a statistical model, which is a complex web of assumptions. This model includes not only the main hypothesis being tested (e.g., a new TweetX campaign has no effect) but also a long list of other assumptions about how the data were generated. These include assumptions about:

• Random sampling or randomization.
• The type of probability distribution the data follow.
• Independence.
• Selection bias.
• The absence of major measurement errors.
• How the analysis was conducted.

A statistical test does not just evaluate the test hypothesis (like the null hypothesis). It evaluates the entire statistical model: the complete set of assumptions. And that’s why we try to always make sure that all other assumptions are fully met and the experiment designs are not flawed: so if we observe a small p-value, we can reasonably read it as evidence against the null, not as proof or ‘acceptance’ of H1.

The P-value is a measure of compatibility, not truth.

The most common definition of a P-value is flawed. A more accurate and useful definition is:

The P-value is a continuous measure of the compatibility between the observed data and the entire statistical model used to compute it. It is the probability that the chosen test statistic would be at least as large as its observed value if every single assumption in the model (including the test hypothesis) were correct.

Think of it as a “surprise index.”

• A small P-value (e.g., P=0.01) indicates that the data are surprising if the entire model were true. It’s a red flag telling us that one or more of our assumptions might be wrong. However, it does not tell us which assumption is wrong. The issue could be the null hypothesis, but it could also be a violated study protocol, selection bias, or another unmet assumption.
• A large P-value (e.g., P=0.40) indicates that the data are not unusual or surprising under the model. It suggests the data are compatible with the model, but it does not prove the model or the test hypothesis is true. The data could be equally compatible with many other models and hypotheses. And that’s why we try to always make sure that all other assumptions are fully met and experiment designs are not flawed: so if we observe a small p-value, we can reasonably read it as evidence against the null, not as proof or ‘acceptance’ of H1.

The common practice of degrading the P-value into a simple binary, “statistically significant” (P≤0.05) or “not significant” (P>0.05), is damaging. It creates a false sense of certainty and ignores the actual quantity of evidence.

Confidence Intervals (CI) and their importance in the Quasi Geo-Lift test.

A confidence interval (CI) is more informative than a simple P-value from a null hypothesis test. It can be understood as the range of effect sizes that are relatively compatible with the data, given the statistical model.

A 95% confidence interval has a specific frequentist property: if you were to repeat a quasi Geo-Lift study countless times with valid statistical models, 95% of the calculated confidence intervals would, on average, contain the true effect size.

Crucially, this does not mean there is a 95% probability that your specific interval contains the true effect. Once calculated, your interval either contains the true value or it doesn’t (0% or 100%). The “95%” tells how often this method would capture the true effect over many repeated studies, not how certain we are about this single interval. If you want to move from frequentist confidence intervals to direct probabilities about lift, Bayesian methods are the way.

If you’d like to dive deeper into p-values, confidence intervals, and hypothesis testing, I recommend these 2 well-known papers:

• A Dirty Dozen: Twelve P-Value Misconceptions – link here.
• Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations – link here.

(E) EVALUATE your Quasi Geo-Lift Experiment with a broader lense.

Remember to zoom out and see the forest, not just the trees. 

The results must feed back into marketing triangulation by validating attribution ROAS, and calibrating marketing mix models with a causal multiplier. 

They should also guide the next steps: replicate positive results in new geographies, assess percentage lift against the minimum detectable threshold, and avoid generalizing from a single market before further testing. 

Stress-testing with placebo testing (in-space or in-time) will also strengthen confidence in your findings.

Below are results from one in-time placebo: +1.3% lift from a placebo is not stat. strong to reject no difference between treatment and control group = H0 (as expected for a placebo):

Incorporating the channel impressions and spend into cross-channel interactions in the MMM helps capture interactions with other channels in your media mix.

If the campaign fails to deliver the expected lift despite planning suggesting it should, it is important to evaluate factors not captured in the quantitative results: the messaging or creative execution. Often, the shortfall may be attributed to how the campaign (doesn’t) resonate with the target audience rather than flaws in the experimental design.

What’s in it for you?

Quasi geo-lift lets you prove whether a campaign truly moves the needle without user-level tracking or big randomized tests. You pick a few markets, build a synthetic control from the rest, and read the incremental impact directly in business units (rides, sales, leads). The ABCDE plan makes it practical:

• Assess marketing triangulation and how you’ll measure,
• Budget to a clear MDE,
• Construct treatment/control and guardrails,
• Deliver ATT → CPIC → profit, then
• Evaluate by calibrating MMM/MTA, stress-testing with placebos, and by looking at your business context with a broader lense.

Net result? Faster, cheaper, defensible answers you can act on.

Thank you for reading. If you enjoyed this article or learned something new, feel free to connect and reach out to me on LinkedIn.

library(tidyr)
library(dplyr)
library(GeoLift)

# Assuming long_data is your pre-formatted dataset with columns: date, location, Y
# The data should be loaded into your R environment before running this code.

long_data <- read.csv("/Users/tomasjancovic/Downloads/long_data.csv")

# Market selection (power analysis)
GeoLift_PreTest <- long_data
GeoLift_PreTest$date <- as.Date(GeoLift_PreTest$date)

# using data up to 2023-09-18 (day before launch)
GeoTestData_PreTest <- GeoDataRead(
  data = GeoLift_PreTest[GeoLift_PreTest$date < '2023-09-18', ],
  date_id = "date",
  location_id = "location",
  Y_id = "Y",
  format = "yyyy-mm-dd",
  summary = TRUE
)

# overview plot
GeoPlot(GeoTestData_PreTest, Y_id = "Y", time_id = "time", location_id = "location")

# power analysis & market selection
MarketSelections <- GeoLiftMarketSelection(
  data = GeoTestData_PreTest,
  treatment_periods = c(14, 21, 28, 35, 42),
  N = c(1, 2, 3, 4, 5),
  Y_id = "Y",
  location_id = "location",
  time_id = "time",
  effect_size = seq(0, 0.26, 0.02),
  cpic = 6,
  budget = 5000,
  alpha = 0.05,
  fixed_effects = TRUE,
  side_of_test = "one_sided"
)

print(MarketSelections)
plot(MarketSelections, market_ID = 4, print_summary = TRUE)

# ------------- simulation starts, you would use your observed treatment/control groups data instead

# parameters
treatment_cities <- c("Zabrze", "Szczecin", "Czestochowa")
lift_magnitude <- 0.11
treatment_start_date <- as.Date('2023-09-18')
treatment_duration <- 21
treatment_end_date <- treatment_start_date + (treatment_duration - 1)

# extending the time series
extend_time_series <- function(data, extend_days) {
  extended_data <- data.frame()
  
  for (city in unique(data$location)) {
    city_data <- data %>% filter(location == city) %>% arrange(date)
    
    baseline_value <- mean(tail(city_data$Y, 30))
    
    recent_data <- tail(city_data, 60) %>%
      mutate(dow = as.numeric(format(date, "%u")))
    
    dow_effects <- recent_data %>%
      group_by(dow) %>%
      summarise(dow_multiplier = mean(Y) / mean(recent_data$Y), .groups = 'drop')
    
    last_date <- max(city_data$date)
    extended_dates <- seq(from = last_date + 1, by = "day", length.out = extend_days)
    
    extended_values <- sapply(extended_dates, function(date) {
      dow <- as.numeric(format(date, "%u"))
      multiplier <- dow_effects$dow_multiplier[dow_effects$dow == dow]
      if (length(multiplier) == 0) multiplier <- 1
      
      value <- baseline_value * multiplier + rnorm(1, 0, sd(city_data$Y) * 0.1)
      max(0, round(value))
    })
    
    extended_data <- rbind(extended_data, data.frame(
      date = extended_dates,
      location = city,
      Y = extended_values
    ))
  }
  
  return(extended_data)
}

# extending to treatment_end_date
original_end_date <- max(long_data$date)
days_to_extend <- as.numeric(treatment_end_date - original_end_date)

set.seed(123)
extended_data <- extend_time_series(long_data, days_to_extend)

# Combining original + extended
full_data <- rbind(
  long_data %>% select(date, location, Y),
  extended_data
) %>% arrange(date, location)

# applying treatment effect
simulated_data <- full_data %>%
  mutate(
    Y_original = Y,
    Y = if_else(
      location %in% treatment_cities &
        date >= treatment_start_date &
        date <= treatment_end_date,
      Y * (1 + lift_magnitude),
      Y
    )
  )

# Verifying treatment (prints just the table)
verification <- simulated_data %>%
  filter(location %in% treatment_cities,
         date >= treatment_start_date,
         date <= treatment_end_date) %>%
  group_by(location) %>%
  summarize(actual_lift = (mean(Y) / mean(Y_original)) - 1, .groups = 'drop')

print(verification)

# building GeoLift input (simulated)
GeoTestData_Full <- GeoDataRead(
  data = simulated_data %>% select(date, location, Y),
  date_id = "date",
  location_id = "location",
  Y_id = "Y",
  format = "yyyy-mm-dd",
  summary = TRUE
)

# Computing time indices
date_sequence <- seq(from = min(full_data$date), to = max(full_data$date), by = "day")
treatment_start_time <- which(date_sequence == treatment_start_date)
treatment_end_time <- which(date_sequence == treatment_end_date)

# Running GeoLift
GeoLift_Results <- GeoLift(
  Y_id = "Y",
  data = GeoTestData_Full,
  locations = treatment_cities,
  treatment_start_time = treatment_start_time,
  treatment_end_time = treatment_end_time,
  model = "None",
  fixed_effects = TRUE
)

# ---------------- simulation ends!

# plots
summary(GeoLift_Results)
plot(GeoLift_Results)
plot(GeoLift_Results, type = "ATT")

# placebos
set.seed(42)

# window length (days) of the real treatment
window_len <- treatment_end_time - treatment_start_time + 1

# the furthest you can shift back while keeping the full window inside pre-period
max_shift <- treatment_start_time - window_len
n_placebos <- 5

random_shifts <- sample(1:max(1, max_shift), size = min(n_placebos, max_shift), replace = FALSE)

placebo_random_shift <- vector("list", length(random_shifts))
names(placebo_random_shift) <- paste0("Shift_", random_shifts)

for (i in seq_along(random_shifts)) {
  s <- random_shifts[i]
  placebo_random_shift[[i]] <- GeoLift(
    Y_id = "Y",
    data = GeoTestData_Full,
    locations = treatment_cities,
    treatment_start_time = treatment_start_time - s,
    treatment_end_time   = treatment_end_time   - s,
    model = "None",
    fixed_effects = TRUE
  )
}

# --- Print summaries for each random-shift placebo ---
for (i in seq_along(placebo_random_shift)) {
  s <- random_shifts[i]
  cat("n=== Summary for Random Shift", s, "days ===n")
  print(summary(placebo_random_shift[[i]]))
}

# Plot ATT for each random-shift placebo
for (i in seq_along(placebo_random_shift)) {
  s <- random_shifts[i]
  placebo_end_date <- treatment_end_date - s
  cat("n=== ATT Plot for Random Shift", s, "days ===n")
  print(plot(placebo_random_shift[[i]], type = "ATT", treatment_end_date = placebo_end_date))
}