Analyze A/B Test Results¶
Table of Contents¶
Introduction¶
A/B tests are very commonly performed by data analysts and data scientists.
For this project, I will be working to understand the results of an A/B test run by an e-commerce website. my goal is to work through this notebook to help the company understand if they should implement the new page, keep the old page, or perhaps run the experiment longer to make their decision.
Analysis was conducted using 3 approaches - probabilistic reasoning, hypothesis testing and logistic regression to cross examine the results produced and draw conclusions.
Tool: Jupter Notebook (Python)
Programming library: panda, numpy, matplotlib, statsmodels
Part I - Probability¶
To get started, let’s import our libraries.
import pandas as pd
import numpy as np
import random
import matplotlib.pyplot as plt
%matplotlib inline
random.seed(42)
1. Now, read in the ab_data.csv data. Store it in df.
a. Read in the dataset and take a look at the top few rows here:
df = pd.read_csv('ab_data.csv')
df.head()
b. Use the cell below to find the number of rows in the dataset.
df.shape[0]
c. The number of unique users in the dataset.
df.nunique()
d. The proportion of users converted.
df.converted.mean()
e. The number of times the new_page and treatment don’t match.
mismatch = df[((df.group == 'treatment') & (df.landing_page!='new_page')) | ((df.group == 'control') & (df.landing_page != 'old_page'))]
mismatch.count()[0]
f. Do any of the rows have missing values?
df.isna().sum()
2. For the rows where treatment does not match with new_page or control does not match with old_page, we cannot be sure if this row truly received the new or old page.
df2 = df.drop(mismatch.index)
# Double Check all of the correct rows were removed - this should be 0
df2[((df2['group'] == 'treatment') == (df2['landing_page'] == 'new_page')) == False].shape[0]
a. How many unique user_ids are in df2?
df2.nunique()
b. There is one user_id repeated in df2. What is it?
duplicated_user=df2[df2.user_id.duplicated()]
duplicated_user
c. What is the row information for the repeat user_id?
duplicated_user
d. Remove one of the rows with a duplicate user_id
df2.drop_duplicates('user_id',inplace = True)
What is the probability of an individual converting regardless of the page they receive?
df2.converted.mean()
b. Given that an individual was in the control group, what is the probability they converted?
obs_p_old=df2[df2['group']=='control'].converted.mean()
c. Given that an individual was in the treatment group, what is the probability they converted?
obs_p_new= df2[df2['group']=='treatment'].converted.mean()
obs_diffs=obs_p_new-obs_p_old
d. What is the probability that an individual received the new page?
df2[df2['landing_page']=='new_page'].count()[0]/df2.count()[0]
Is there a sufficient evidence to conclude that the new treatment page leads to more conversions?
There is no sufficient evidence as the mean of conversion regardless of groups is the same of the mean in treatment group who recieved the new page furthermore the mean in control group is more than the mean in treatment
Part II - A/B Test¶
Notice that because of the time stamp associated with each event, you could technically run a hypothesis test continuously as each observation was observed.
However, then the hard question is do you stop as soon as one page is considered significantly better than another or does it need to happen consistently for a certain amount of time? How long do you run to render a decision that neither page is better than another?
These questions are the difficult parts associated with A/B tests in general.
1. For now, consider I need to make the decision just based on all the data provided. If I want to assume that the old page is better unless the new page proves to be definitely better at a Type I error rate of 5%, what should null and alternative hypotheses be?
null hypothesis is
$p_{new}$ - $p_{old}$ < 0
alternative hypothesis is
$p_{new}$ - $p_{old}$ > 0
2. Assume under the null hypothesis, $p_{new}$ and $p_{old}$ both have “true” success rates equal to the converted success rate regardless of page - that is $p_{new}$ and $p_{old}$ are equal. Furthermore, assume they are equal to the converted rate in ab_data.csv regardless of the page.
Use a sample size for each page equal to the ones in ab_data.csv.
Perform the sampling distribution for the difference in converted between the two pages over 10,000 iterations of calculating an estimate from the null.
a. What is the conversion rate for $p_{new}$ under the null?
p_new= df2.converted.mean()
p_new
b. What is the conversion rate for $p_{old}$ under the null?
p_old= df2.converted.mean()
p_old
c. What is $n_{new}$, the number of individuals in the treatment group?
n_new= df2.query('group== "treatment" ').count()[0]
n_new
d. What is $n_{old}$, the number of individuals in the control group?
n_old= df2.query('group== "control"').count()[0]
n_old
e. Simulate $n_{new}$ transactions with a conversion rate of $p_{new}$ under the null. Store these $n_{new}$ 1’s and 0’s in new_page_converted.
new_page_converted= np.random.binomial(n_new, p_new)
f. Simulate $n_{old}$ transactions with a conversion rate of $p_{old}$ under the null. Store these $n_{old}$ 1’s and 0’s in old_page_converted.
old_page_converted = np.random.binomial(n_old, p_old)
g. Find $p_{new}$ - $p_{old}$ for simulated values from part (e) and (f).
new_page_converted/n_new - old_page_converted/n_old
h. Create 10,000 $p_{new}$ - $p_{old}$ values using the same simulation process used in parts (a) through (g) above. Store all 10,000 values in a NumPy array called p_diffs.
p_diffs = []
for _ in range(10000):
new_page_converted= np.random.binomial(n_new, p_new)
old_page_converted = np.random.binomial(n_old, p_old)
p_diffs.append(new_page_converted/n_new - old_page_converted/n_old)
p_diffs = np.array(p_diffs)
i. Plot a histogram of the p_diffs.
plt.hist(p_diffs);
j. What proportion of the p_diffs are greater than the actual difference observed in ab_data.csv?
(p_diffs>obs_diffs).mean()
This value called p value, it means that under null hypothesis there is 10% chance for difference of mean to be as we observed ,and that mean there is no evidence to reject null hypothesis
calculate the number of conversions for each page, as well as the number of individuals who received each page. n_old and n_new refer the the number of rows associated with the old page and new pages, respectively.
import statsmodels.api as sm
convert_old = df2.query('group== "treatment"').converted.sum()
convert_new = df2.query('group== "control"').converted.sum()
n_old = df2.query('group== "control"').count()[0]
n_new = df2.query('group== "treatment" ').count()[0]
Now use stats.proportions_ztest to compute your test statistic and p-value.
sm.stats.proportions_ztest([convert_new, convert_old], [n_new, n_old], alternative='smaller')
z score mean that observed conversion rate is 1 standard deviation apart from the mean
Part III - A regression approach¶
1. In this final part, we will see that the result we achieved in the A/B test in Part II above can also be achieved by performing regression.
a. Since each row is either a conversion or no conversion,we will perform logistic regression
b. The goal is to use statsmodels to fit the regression model to see if there is a significant difference in conversion based on which page a customer receives. However, we first need to create in df2 a column for the intercept, and create a dummy variable column for which page each user received. Add an intercept column, as well as an ab_page column, which is 1 when an individual receives the treatment and 0 if control.
df2['ab_page'] = pd.get_dummies(df2['group'])['treatment']
df2['intercept'] = 1
c. Use statsmodels to instantiate our regression model on the two columns we created in part b., then fit the model using the two columns we created in part b. to predict whether or not an individual converts.
log_mod = sm.Logit(df2['converted'], df2[['intercept','ab_page' ]])
results = log_mod.fit()
d. The summary of the model
results.summary()
e. What is the p-value associated with ab_page?
p value= .190 ,it differ from the value found in Part II because this is 2 tailed test but in part 2 is is one tailed
we will add other variables to try to explain as much variation in convert as possible, potential problems are Non-linearity of the response-predictor relationships
Correlation of error terms
Non-constant Variance and Normally Distributed Errors
Outliers/ High leverage points
Multicollinearity
Does it appear that country had an impact on conversion?
countries=pd.read_csv('countries.csv')
df3=df2.set_index('user_id').join(countries.set_index('user_id'))
df3['CA'] = pd.get_dummies(df3['country'])['CA']
df3['UK']= pd.get_dummies(df3['country'])['UK']
log_mod = sm.Logit(df3['converted'], df3[['intercept','CA','UK','ab_page' ]])
results = log_mod.fit()
results.summary()
from p value it’s appear that country has no impact on conversion
we would now like to look at an interaction between page and country to see if there significant effects on conversion. Create the necessary additional columns, and fit the new model.
df3["CA_page"] = df3["ab_page"]*df3["CA"]
df3["UK_page"] = df3["ab_page"]*df3["UK"]
log_mod = sm.Logit(df3['converted'], df3[['intercept','CA_page','UK_page' ]])
results = log_mod.fit()
results.summary()
from subprocess import call
call(['python', '-m', 'nbconvert', 'Analyze_ab_test_results_notebook.ipynb'])