Class 19 Natural Experiment II: Difference-in-Differences Design
1 Difference-in-Differences Design
1.1 Learning Objectives
Understand the intuition of the difference-in-differences (DiD) design.
Learn how to estimate the treatment effect using DiD design.
The intuition of Synthetic Difference-in-Differences when the parallel pre-trend assumption is violated.
1.2 Core Business Question
Marks and Spencer is looking to investigate the causal effect of removing delivery fee on customer total spending.
- What are the problems of using A/B testing?
Spill-over effects: Customers in the control group may learn about the free delivery policy from customers in the treatment group, leading to contamination of the control group.
Public relations issues: Customers in the control group may feel unfairly treated, leading to negative publicity and customer dissatisfaction.
1.3 City-Level Pilot Test
- Due to these concerns, M&S decides to implement a city-level pilot test: free delivery is offered to all customers in Cambridge (treatment group) starting in 1 December, while customers in other cities continue to pay delivery fees (control group). And they collect data on customer spending in November and December:
| November | December | |
|---|---|---|
| Cambridge (Treatment) | 160 | 200 |
| Other Cities (Control) | 140 | 165 |
Can we compare the changes in Cambridge as the causal effect: £40 (=200-160)
Can we compare Cambridge with other cities in December as the causal effect: £35 (=200-165)?
- Neither! £40 ignores the fact that customer spending may increase in December due to seasonal effects (e.g., holiday shopping). £35 ignores the fact that customer spending in Cambridge may have a different baseline spending compared to other cities.
1.4 Introduction to DiD
- Difference-in-Differences Design (DiD, DD, Diff-in-Diff) is a statistical technique used in economics and business that attempts to mimic an experimental research design using secondary data by comparing the changes in the outcomes of the treatment group with the changes in the outcomes of the control group.
1.5 Requirements for Using Difference-in-Differences
Control group: We need to find a group of units who are unaffected by the natural experiment
Treatment group: We need to find a treatment group of units who are affected by the natural experiment
No cross-over and spill-over (SUTVA): There is no interference between the treatment and control group that can cause cross-over or spill-over effects.1
Parallel trends assumption: The treatment and control groups must have parallel trends before treatment occurs.2
1.6 Parallel Trends Assumption
- Counterfactual Reasoning: To estimate the causal effect, we need to compare the observed outcome of the treated group with their counterfactual outcome (what would have happened if they had not been treated).
- Cambridge’s total spending change is 200 - 160 = 40, but we need to know what would have happened to Cambridge’s total spending if there had been no free delivery policy.
- Imputing the Counterfactual: Since the counterfactual is unobservable, DiD uses the trend of the control group to impute the counterfactual trend of the treatment group. The assumption implies that, in the absence of treatment, the average outcomes of the treated and control groups would have followed parallel paths over time.
- Other cities’ total spending change is 165 - 140 = 25, so we assume that Cambridge’s total spending would also have increased by only 25 if there had been no free delivery policy.
- Consequence: If the assumption holds, any deviation from the parallel path in the post-treatment period can be attributed to the causal effect of the treatment.
- Therefore, the causal effect of free delivery on customer spending in Cambridge is 40 - 25 = 15.
- Since the treatment effect is derived for the treated group data, DiD estimates the Average Treatment Effect on the Treated (ATT). Whether or not the treatment effect can be generalized to the whole population depends on the context.
2 Implementation of DiD Using R
2.1 DiD Estimation: Linear Regression
\[ Outcome_{i, t}=\beta_0+ \beta_{1} Post_{t}+\beta_{2} Treated_{i}+\beta_{3} Treated_{i} \times Post_{t} + \mu_{i, t} \]
- \(Post_{t}\) controls for the seasonality for both treatment and control groups
- \(Treated_{i}\) controls for the pre-existing difference between the treatment group and control group.
- After accounting for (1) seasonality (\(\beta_1\)) and (2) pre-existing across-group differences (\(\beta_2\)), the interaction term (\(\beta_3\)) measures the treatment effect.
2.2 Application of DiD: The Causal Effect of Privacy Regulation
Firms routinely collect consumer data on mobile apps and develop algorithms to target customers with personalised ads. However, consumers increasingly value privacy as an intrinsic human right: “the right to be left alone”.
Regulators and mobile ecosystems have enacted various regulations to restrict firms’ collection of sensitive customer data.
- EU’s GDPR (2018), California’s CCPA (2020), China’s PIPL (2022)
- Apple’s App Tracking Transparency (2021), Android’s Privacy Sandbox
It’s important to understand the causal effects of the privacy regulations on firms and consumers: (1) Trust-enhancing effect (2) Efficiency-decreasing effect
2.3 Apple’s App Tracking Transparency Policy on Consumer Spending
Before iOS 14.5 (26 April 2021), user data tracking on iOS lacked explicit user consent: iOS apps and advertisers could track user activities across different apps through the Identifier for Advertisers (IDFA).
After iOS 14.5, Apple introduced the App Tracking Transparency (ATT) policy, which requires apps to obtain explicit user consent before tracking user activities across different apps.
Causal Question: How does the implementation of Apple’s App Tracking Transparency Policy affect consumer spending at your business?
2.4 The Causal Effect of the Introduction of ATT Policy
- \(y\): standardised customer spending.
- \(x1\): standardised customer income.
- \(id\): Identifier of the customer.
- \(period\): normalised time period. 0 is the month in which the ATT policy was introduced.
- \(post\): equals 1 for after ATT was introduced.
- \(treat\): equals 1 for iPhone customers.
2.5 Data Preprocessing
When you run DiD analysis, you need to construct a panel dataset similar to the following one.
If the raw data are transaction data, you need to aggregate the data at the unit-time level, such as customer-month level.
2.6 Estimation of DiD Using Linear Regressions
- We need to run a linear regression with three variables:
treat,post, and the interaction termtreat * post\[ Outcome_{i, t}=\beta_0+ \beta_{1} post_{t}+\beta_{2} treat_{i}+\beta_{3} treat_{i} \times post_{t} + \mu_{i, t} \]
2.7 Report the DiD Results
| (1) | |
|---|---|
| + p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001 | |
| (Intercept) | 0.323 |
| (0.317) | |
| treat | 0.142 |
| (0.445) | |
| post | 0.713 |
| (0.449) | |
| treat × post | 4.993*** |
| (0.629) | |
| Num.Obs. | 1080 |
| R2 | 0.183 |
- The coefficient of the interaction term
treat:postis the treatment effect of the ATT policy. - After introducing the ATT policy, iPhone users increased their spending by 4.993 units compared to Android users.
2.8 Testing the Parallel Pre-trend Assumption
- Before we draw the causal conclusion, we must test the parallel pre-trend assumption by plotting the average outcome for the treatment and control groups over time.
Code
pacman::p_load(dplyr, ggplot2, ggthemes)
group_did <- base_did %>%
group_by(treat, period) %>%
summarise(avg_y = mean(y, na.rm = T)) %>%
ungroup()
ggplot() +
geom_line(
data = group_did,
aes(
x = period,
y = avg_y,
color = factor(treat)
)
) +
scale_x_continuous(breaks = unique(base_did$period)) +
theme_stata()
- The graph indicates a violation of the parallel pre-trend assumption.
3 Synthetic DiD (Optional)
3.1 When the Parallel Pre-Trend is Violated
If the parallel pre-trend assumption is violated, we can use synthetic difference-in-differences, which is a method that combines synthetic control and difference-in-differences.
The Synthetic Control Method is a method that uses unit weighting to create a synthetic control group that approximates the pre-treatment outcomes of the treatment group (Abadie, Diamond, and Hainmueller 2010).
However, Synthetic Control is very restrictive in many ways. Most importantly, it forces the treatment group to have the same level of the outcome variable as the synthetic control group, which is a very strong restriction.
3.2 Synthetic Difference-in-Differences
- Synthetic Difference-in-Differences is a method that uses synthetic control methods to estimate the treatment effect when the parallel pre-trend assumption is violated (Arkhangelsky et al. 2021; Clarke et al. 2023).
Compared with the Synthetic Control Method, Synthetic DiD is more flexible and allows the treatment group to have different levels of the outcome variable.
Compared with the DiD method, Synthetic DiD can estimate the treatment effect when the parallel pre-trend assumption is violated.
3.3 Prepare the Data for Synthetic DiD
- We need to prepare the data for the synthetic DiD method to the required format.
3.4 Run SynthDiD
- After the dataset is prepared according to the
panel.matricesfunction, we can run thesynthdid_estimate()function to estimate the treatment effect.
3.5 Visualisation of SynthDiD
- Within the synthdid package, the
plot()function can be used to visualise the results of the synthetic DiD method.
3.6 References
Abadie, Alberto, Alexis Diamond, and Jens Hainmueller. 2010. “Synthetic Control Methods for Comparative Case Studies: Estimating the Effect of California’s Tobacco Control Program.” Journal of the American Statistical Association 105 (490): 493–505. https://doi.org/10.1198/jasa.2009.ap08746.
Arkhangelsky, Dmitry, Susan Athey, David A. Hirshberg, Guido W. Imbens, and Stefan Wager. 2021. “Synthetic Difference-in-Differences.” American Economic Review 111 (12): 4088–118. https://doi.org/10.1257/aer.20190159.
Clarke, Damian, Daniel Pailañir, Susan Athey, and Guido Imbens. 2023. “Synthetic Difference In Differences Estimation.” arXiv. https://arxiv.org/abs/2301.11859.