Class 18 Difference-in-Differences Design

Author
Affiliation

Dr Wei Miao

UCL School of Management

Published

November 29, 2023

1 Difference-in-Differences Design

1.1 What Can Go Wrong with RDiT

  • The underlying assumption for RDiT is that, the pre-treatment outcomes before the natural experiment are good counterfactual outcomes for the post-treatment outcomes if the natural experiment had not happened.

\[ Y_{it}=\alpha+\beta_{1} Post_{it} + \mu_{i t} \]

  • However, we cannot guarantee that there is no seasonality. Therefore, we need to find alternative methods that require less strict assumptions than RDiT.

1.2 Difference-in-Differences Design

  • If we can find a control group of individuals who are unaffected by the natural experiment, we can then use difference-in-differences design.

  • Difference-in-differences design (DiD, DD, Diff-in-Diff) is a statistical technique used in economics that attempts to mimic an experimental research design using observational data, by comparing the changes in the outcomes of the treatment group with the changes in the outcomes of the control group.

1.3 Visualization of DiD Design

1.4 DiD Estimator: Alternative Illustration

  • We can use a DiD regression to quantify the treatment effects:
    • \(Post_{t}\) controls for the seasonality for all customers
    • \(Treated_{i}\) controls for the pre-existing difference between the treatment group and control group.

\[ Outcome_{i, t}=\beta_0+ \beta_{1} Post_{t}+\beta_{2} Treated_{i}+\beta_{3} Treated_{i} \times Post_{t} + \mu_{i, t} \]

  • Therefore, after teasing out (1) seasonality \(\beta_1\) and (2) pre-existing across-group differences \(\beta_2\) , \(\beta_3\) measures the actual treatment effect,

1.5 Parallel Pre-trend Assumption

  • The requirement for a valid DiD analysis is that there is no differential trend between the treatment and control group before the treatment happens, or we must need parallel pre-trend.

2 Implementation of DiD Using R

2.1 The Causal Effect of a New Loyalty Program

  • Tesco introduces a new loyalty program in Manchester in June while customers in London still use the old loyalty program.
    • \(y\): standardized customer spending
    • \(x1\): standardized customer income.
    • \(id\): Identifier of the customer.
    • \(period\): January to October
    • \(post\): equals 1 for June and onwards.
    • \(treat\): equals 1 for Manchester customers.
pacman::p_load(fixest,modelsummary,dplyr)
data("base_did")
Tip

When randomization is difficult, you can use DiD design to replace randomized experiments.

2.2 Dataset

head(base_did,10)
y x1 id period post treat
2.8753063 0.5365377 1 1 0 1
1.8606527 -3.0431894 1 2 0 1
0.0941652 5.5768439 1 3 0 1
3.7814749 -2.8300587 1 4 0 1
-2.5581996 -5.0443544 1 5 0 1
1.7287324 -0.6363849 1 6 1 1
6.2842363 -2.1298837 1 7 1 1
4.7668878 3.4918558 1 8 1 1
5.3475339 0.8379861 1 9 1 1
4.9514693 0.6338890 1 10 1 1

2.3 Estimation of DiD

  • We need to run a linear regression with 3 variables: treat, post, and the interaction term

\[ Outcome_{i, t}=\beta_0+ \beta_{1} Post_{t}+\beta_{2} Treated_{i}+\beta_{3} Treated_{i} \times Post_{t} + \mu_{i, t} \]

est_did = feols(
  fml = y ~ treat + post + treat:post, 
                data = base_did)

2.4 Report the DiD Results

 (1)
(Intercept) 0.323
(0.317)
treat 0.142
(0.445)
post 0.713
(0.449)
treat × post 4.993***
(0.629)
Num.Obs. 1080
R2 0.183
+ p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001
  • After the new loyalty program was introduced, compared with the control group, the treatment group customers increase their total spending by 4.993 units.

2.5 Testing the Parallel Pre-trend Assumption

  • The loyalty program was introduced in June, we need to make sure that, the differences across the treatment group and control group are parallel before June. We can visually examine the assumption:
    • Step 1: use dplyr group aggregation to compute the average outcome for each group in each month
    • Step 2: use ggplot2 to visualize the trend.
pacman::p_load(dplyr,ggplot2, ggthemes)

group_did <- base_did%>%
  group_by(treat,period) %>%
  summarise(avg_y = mean(y, na.rm = T))%>%
  ungroup()

ggplot() + 
  geom_line(data = group_did,
            aes(x = period,
                y = avg_y,
                color = factor(treat))) + 
  theme_stata()

2.6 When the Parallel Pre-Trend is Violated