Class 13 Case Study: Improve User Engagement for Instagram Using A/B/N Testing
1 Case Background
1.1 Business Objective
Instagram aims to increase user engagement and activity.
We can propose gamification strategies based on scientific theories.
Need to empirically test whether proposed gamification strategies are effective using A/B/N tests.
2 Theoretical Motivations
2.1 Theoretical Motivation for Business Ideas
When proposing business ideas, we should base our proposals on scientific, well-established theories from different disciplines.
Bottom-up approach: start with established theories and then generate business ideas
Top-down approach: start with business ideas and then find theories to support and explain them for generalisability
Let’s first see some examples of behavioural economics theories!
2.2 Behavioural Theories
2.3 Default Effect
2.4 Left-Digit Bias
2.6 Prospect Theory
Prospect theory posits that people feel more pain from losing something than pleasure from gaining something.
This theory can be used to explain why people are more likely to engage in activities that prevent loss than those that promote gain.
2.7 Business Proposal
- Implement gamification features on Instagram to increase user activity based primarily on Social Comparison Theory.
- Generate ideas that can sustainably boost user engagement while safeguarding user well-being.
2.8 Potential Strategies
Endowment effect: Implement a points and badge system to create a sense of ownership and encourage engagement (e.g. likes, comments, shares).
Social comparison theory: Leaderboards showing top users; activity rankings and comparative progress panels.
Any other ideas?
3 A/B/N Testing for Instagram
3.1 Step 1: Decide on the Unit of Randomisation
- What would be the best unit of randomisation?
- What are the potential problems for spillover and crossover?
Tip
For solutions to this case study, please check the website version of the case study here
3.2 Step 2: Decide on Randomisation Allocation Scheme
- How should we determine the randomisation scheme?
3.3 Step 3: Decide on Sample Selection and Treatment Duration
- What is the sample size we need?
3.4 Step 4: Collect Data
- What data should we collect?
3.5 Step 5: Interpreting Results from a Field Experiment
Randomisation check: Verify treatment and control groups are well balanced on pre-treatment characteristics; adjust with regression if imbalances arise.
Analyse the data and estimate the ATE: Use difference in means (A/B) or regression with treatment indicators (A/B/N) to estimate average treatment effects (ATE), adding covariates only if needed for precision or imbalance correction.
2.5 Social Comparison Theory
People evaluate their own opinions and abilities by comparing themselves to others, especially when comparing oneself to similar others.
Social comparison can be upward or downward.
Social comparison can motivate people to improve their performance; however, it can also lead to negative emotions.