Improve User Engagement for Instagram Using A/B/N Testing

MSIN0094 Case Study

Author

Affiliation

Dr Wei Miao

UCL School of Management

1 Case background

Instagram, one of the world’s leading social media platforms, has achieved significant success by providing users with a visually driven, interactive space for self-expression, connection, and content sharing. With over a billion active users, Instagram has become a major platform for individuals, influencers, and businesses to connect with broader audiences. Despite its popularity, Instagram faces bottlenecks in maintaining high levels of user engagement and growth, particularly as competition from other platforms increases. To address these challenges, Instagram must continuously innovate and find new ways to keep users active and engaged.

Screenshot of A Nobody’s Instagram

Question 1

• What is Instagram’s business model?

• How does Instagram make revenues?

• Who are Instagram’s customers?

• What are the major competitors and their relative strengths and weaknesses compared with Instagram?

• Who are the collaborators of Instagram?

• PESTLE analysis: any particular legal and regulatory issues that Instagram needs to be aware of?

Answer

• What is Instagram’s business model?

  • Platform business model. Network effect is the key to success.

• How does Instagram make revenues?

  • Advertising: Sponsored posts, stories, and videos.

  • E-commerce: Shopping features that allow users to purchase products directly from the platform.

• Who are Instagram’s customers?

  • Users: Individuals who use the platform to share photos, videos, and stories, connect with friends, and discover new content.

  • Advertisers: Brands and businesses that promote their products and services to Instagram’s user base through sponsored content and ads.

  • Content Creators: Influencers, celebrities, and creators who produce engaging content to attract followers and monetize their presence on the platform.

• What are the major competitors and their relative strengths and weaknesses compared with Instagram?

  • Direct: Other social media platforms like Facebook, Tiktok, etc.

  • Indirect: News websites, discussion forums like Reddit, and alternative communication platforms that offer different ways for people to obtain information and interact online.

• Who are the collaborators of Instagram?

  • Business Partners: Companies that integrate Instagram content into their services, such as news organizations or broadcasters.

  • Influencers and celebrities who attract and engage large audiences.

• PESTLE analysis: any particular legal and regulatory issues that Instagram needs to be aware of?

  • Legal and Regulatory Issues: Changes in regulations related to data privacy, online speech, and censorship can significantly impact operations.

Gamification offers a promising approach to deepen engagement and retain users, creating a more interactive and rewarding experience that may align with Instagram’s focus on community building and personal expression. In recent years, gamification has emerged as a powerful tool in marketing and business, aimed at enhancing user engagement and loyalty across various industries. Gamification involves integrating game-like elements—such as points, levels, badges, and rewards—into non-game contexts to make experiences more engaging and enjoyable (Seaborn and Fels 2015). By tapping into fundamental human motivations like achievement, competition, and social interaction, gamification encourages users to interact more frequently and meaningfully with a platform or brand. From loyalty programs to interactive challenges, gamification strategies are designed to increase user activity and foster a deeper emotional connection to the brand.

2 Gamification strategies for Instagram

Social media platforms, in particular, are leveraging gamification to boost user engagement and retention. With features like badges, leaderboards, and interactive challenges, these platforms aim to make user interactions feel rewarding and enjoyable. In this case study, we will explore how gamification can be effectively applied on Instagram, using psychological and behavioral economic theories to design features that drive user engagement.

2.1 Prospect theory

Prospect theory, a behavioral economic theory, describes how people make decisions under uncertainty (Kahneman 1979). It suggests that individuals evaluate potential gains and losses relative to a reference point and tend to be risk-averse in the domain of gains but risk-seeking in the domain of losses.

Prospect Theory

In e-commerce, many platforms use prospect theory by presenting “limited time only” sales or “only 3 items left!” notifications. For instance, an online clothing retailer might display a countdown for flash sales on specific products, encouraging customers to buy now or miss out. Additionally, a feature like “items left in your cart are almost sold out” leverages loss aversion to encourage users to complete their purchases. This taps into the user’s fear of missing out (FOMO), driving them to act quickly to avoid potential regret over a lost deal.

Question 2

Based on the Prospect Theory, how could Instagram leverage the concept of loss aversion to increase user engagement?

Answer

Instagram could introduce a time-limited reward system where users earn exclusive virtual items or features by engaging with the platform within a specific timeframe. For example, users could receive a special badge or virtual currency for posting a certain number of stories or liking a certain number of posts within a week. If users fail to meet the activity goal, they lose the opportunity to earn the reward, creating a sense of loss aversion that motivates them to participate more actively to avoid missing out.

2.2 Social comparison theory

Social comparison theory posits that individuals gauge their social and personal worth by comparing themselves to others (Ye et al. 2022). People engage in social comparison to assess their abilities, opinions, and status relative to others. By creating a leaderboard, users could see their standings based on metrics like follower engagement, post interactions, or content quality. Ranking users against one another could motivate them to post more frequently, improve content quality, or engage more with other users’ posts. The leaderboard would provide real-time feedback, fostering both upward and downward comparisons. Users higher up the leaderboard would feel motivated to maintain their status, while those lower might be motivated to increase their activity to climb the ranks.

Social Comparison Theory

Fitness apps often incorporate social comparison to keep users motivated. For example, a fitness app might display a leaderboard that ranks users by steps taken, calories burned, or workout streaks. Users can compare their progress with friends or a larger community. This often motivates users to stay active and improve their standing. By allowing users to see both how they compare to others and celebrate their achievements, such apps encourage regular engagement and goal completion. Companies like Strava, which adds social aspects to running and cycling, have successfully used this approach to drive sustained engagement.

Question 3

Think about how Instagram could leverage the Social Comparison Theory to increase user engagement. Provide a specific example of a feature or mechanism that Instagram could implement to encourage social comparison among users.

Answer

Instagram could introduce a gamified leaderboard system that ranks users based on their daily activity levels, such as the number of posts, likes, comments, or stories shared. Users could see their rankings relative to other users, encouraging competition and social comparison. For example, Instagram could display a daily leaderboard that shows the top users based on their activity metrics, motivating others to increase their engagement to climb the rankings. This feature would leverage social comparison theory by creating a sense of competition and achievement among users, driving them to engage more actively with the platform.

3 Testing proposals using A/B/N testing

We have proposed various strategies to boost user activity on Instagram. Now, we need to design an A/B/N testing plan to evaluate the effectiveness of these strategies.

3.1 Step 1: Decide on the Unit of Randomization

Question 4

• What would be the best unit of randomization?

Answer

• The ideal randomization level would be the user level.

• Device level would be too granular and can easily cause crossover effects.

• Need to force users to log in using the same account to make sure there is no crossover effect. This explains why websites and apps always ask users to log in before using the service.

3.2 Step 2: Mitigate Spillover and Crossover Effects

Question 5

• What are the potential problems for spillover and crossover?

Answer

• A user may use multiple devices, causing crossover effects; that is, the same user may be exposed to different treatments on their phones, laptops, and tablets.

  • This can be mitigated by forcing users to log in using the same account on all devices.

• Spillover effects may occur when a user talks to family members or friends about the treatment they received, potentially influencing their behavior as well as the user’s. Meanwhile, even if the user does not directly talk to others about the treatment, they may still influence others’ behavior through their actions on the platform due to the network effect.

  • This can be mitigated by ensuring that users are not aware of the treatment they received and by keeping the treatment confidential.

3.3 Step 3: Decide on Randomization Allocation Scheme

Question 6

• How should we determine the randomization scheme?

Answer

• Since A/B/N testing can be costly and risky, normally we would not use all the users.

  • Method 1: On testing launch date, we can randomly assign users to different treatment groups based on their user ID. For instance, we can assign 10% of users to the treatment group and 90% to the control group. Then, we can take the last digit of the user ID and assign the user to the treatment group if the last digit is 0, and to the control group if the last digit is 1-9.

  • Method 2: We can also randomly assign users to different treatment groups based on random sampling. See the codes below.

• After randomization is assigned, the treatment should remain the same for each user during the experiment period.

# Method 1: Randomization using user ID

data_user <- data_user %>%
    mutate(treated = ifelse(ID %% 10 == 0, 1, 0 ))

# Method 2: Randomization using R

# how to randomize the treatment if there is 1 control group and 1 treatment group

set.seed(888)

# assign 10% of users to the treatment group
treatment_probability <- 0.1

treated_index <- sample(1:nrow(data_user),
    nrow(data_user) * treatment_probability,
    replace = F
)

data_user <- data_user %>%
    mutate(treated = ifelse(ID %in% treated_index,
        1,
        0
    ))

3.4 Step 4: Collect Data

Question 7

• What is the sample size we need?

• What data should we collect?

Answer

• We can do a power analysis using pwr package in R, or simply some websites, e.g., this link.

• We need to collect the following two types of data. The data serve 2 purposes: (1) randomization check (2) estimation of treatment effects

  • Demographic data, this helps us to conduct the randomization check.

  • Behavioral data, this helps us to estimate the treatment effects.

3.5 Step 5: Data analytics

Question 8

• Once data are collected, how can we test our hypothesis?

Answer

• First, we need to do a randomization check to ensure that the treatment group and control group users have similar characteristics. For any significant differences, we need to run a regression model to control for these differences.

pacman::p_load(dplyr)
data_instagram <- read.csv("https://www.dropbox.com/scl/fi/wf7al7k8go8tg9rkf033r/instagram_ab_test.csv?rlkey=5u43dxj705iepx1bir5h7snkg&dl=1")


# examine if there is any difference across the treatment and control groups
t.test(age ~ treatment,
    data = data_instagram %>%
    filter(treatment %in% c('control','A'))
)

    Welch Two Sample t-test

data:  age by treatment
t = -0.90268, df = 636.35, p-value = 0.367
alternative hypothesis: true difference in means between group A and group control is not equal to 0
95 percent confidence interval:
 -1.1134248  0.4121476
sample estimates:
      mean in group A mean in group control 
             24.91722              25.26786 

t.test(age ~ treatment,
    data = data_instagram %>%
    filter(treatment %in% c('control','B'))
)

    Welch Two Sample t-test

data:  age by treatment
t = -0.61826, df = 668.76, p-value = 0.5366
alternative hypothesis: true difference in means between group B and group control is not equal to 0
95 percent confidence interval:
 -0.9547753  0.4974924
sample estimates:
      mean in group B mean in group control 
             25.03922              25.26786 

• Next, we can analyze the treatment effects by comparing the key activity metrics between the treatment and control groups. We can use pairwise t-tests if it’s A/B testing, or linear regression models if it’s A/B/N testing.

data_instagram_avg <- data_instagram %>%
    group_by(treatment) %>%
    summarise(avg_post_total_activity = mean(post_total_activity)) %>%
    ungroup()

# compare the treatment effects for proposal A versus control
data_instagram_avg$avg_post_total_activity[1] - data_instagram_avg$avg_post_total_activity[3]

[1] 13.3913

# compare the treatment effects for proposal B versus control
data_instagram_avg$avg_post_total_activity[2] - data_instagram_avg$avg_post_total_activity[3]

[1] 33.70592

# is the difference statistically significant?
t.test(post_total_activity ~ treatment,
    data = data_instagram %>%
    filter(treatment %in% c('control','A'))
)

    Welch Two Sample t-test

data:  post_total_activity by treatment
t = 24.481, df = 586.29, p-value < 2.2e-16
alternative hypothesis: true difference in means between group A and group control is not equal to 0
95 percent confidence interval:
 12.31696 14.46564
sample estimates:
      mean in group A mean in group control 
             56.18212              42.79082 

t.test(post_total_activity ~ treatment,
    data = data_instagram %>%
    filter(treatment %in% c('control','B'))
)

    Welch Two Sample t-test

data:  post_total_activity by treatment
t = 57.328, df = 550.65, p-value < 2.2e-16
alternative hypothesis: true difference in means between group B and group control is not equal to 0
95 percent confidence interval:
 32.55102 34.86081
sample estimates:
      mean in group B mean in group control 
             76.49673              42.79082 

• We will see that, we can also run a linear regression to obtain the average treatment effects for A/B/N testings.

pacman::p_load(modelsummary, fixest)
# run a linear regression model to estimate the treatment effects

# create dummy variables for treatment groups

data_instagram <- data_instagram %>%
mutate(treatment_factor = as.factor(treatment)) %>%
mutate(treatment_factor = relevel(treatment_factor, ref = "control"))

feols(fml = post_total_activity ~ treatment_factor,
data = data_instagram) %>%
modelsummary(stars = TRUE)

	(1)
+ p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001
(Intercept)	42.791***
	(0.379)
treatment_factorA	13.391***
	(0.575)
treatment_factorB	33.706***
	(0.573)
Num.Obs.	1000
R2	0.777
R2 Adj.	0.776
AIC	6872.4
BIC	6887.1
RMSE	7.50
Std.Errors	IID

Based on the analyses, it seems that both proposal A and proposal B have a significant positive impact on user engagement. However, we need to consider the costs and feasibility of implementing these features on Instagram. By conducting A/B/N testing, we can evaluate the effectiveness of different gamification strategies and make data-driven decisions to optimize user engagement on the platform.

References

Kahneman, Daniel. 1979. “Prospect Theory: An Analysis of Decisions Under Risk.” Econometrica 47: 278. https://doi.org/10.2307/1914185.

Seaborn, Katie, and Deborah I Fels. 2015. “Gamification in Theory and Action: A Survey.” International Journal of Human-Computer Studies 74: 14–31.

Ye, Teng, Wei Ai, Yan Chen, Qiaozhu Mei, Jieping Ye, and Lingyu Zhang. 2022. “Virtual Teams in a Gig Economy.” Proceedings of the National Academy of Sciences 119 (51): e2206580119. https://doi.org/10.1073/pnas.2206580119.