9: Hypothesis Testing

State the Hypothesis

Since the yes/no question we try to answer has two results, such as win/loss data when a sport team plays at home and win/loss data when the team plays away from home, hypothesis testing includes two hypotheses:

1. The null hypothesis says that there is no significant difference between the two results. Any difference is due to random chance rather than due to the influence of a real phenomenon.
   The term null means that any difference is due to nothing but chance.
2. The alternative hypothesis says that any difference between the two results is due to some influence rather than chance.

For the coin toss experiment:

1. The null hypothesis: The coin is fair, there is no influence making heads or tails more likely. P(heads) is 0.5.
2. The alternative hypothesis: The coin is not fair and something influences it to land on one side more often. P(heads) is not 0.5.

Build a Model

The model shows what the data would look like if the null hypothesis is true.

For the coin toss experiment, we build a model that shows the likely result if the coin is fair. But before we use the model for hypothesis testing, it is important to choose a model and assess that it is valid for our data.

import random
import numpy as np

# coin_toss_simulation has 1 input: the number_of_times the coin is to be tossed
def coin_toss_simulation(number_of_times):
  # loop number_of_times to tossing a coin number_of_times
  L = []
  for i in range(number_of_times):
    rand_num = random.randint(0,1)
    L.append(rand_num)
  outcome = np.array(L)
  return outcome    # outcome is a numpy array of 0s and 1s, where 0 = heads and 1 = tails

# run the simulation 10,000 times, each time the coin is tossed 500 times
total_number_of_heads = []
for i in range(10000):
  # get array of 0s and 1s from tossing a coin 500 times
  results = coin_toss_simulation(500)
  # get the number of heads from the results
  number_of_heads = np.sum(results == 0)
  # save the number of heads to the total_number_of_heads list
  total_number_of_heads.append(number_of_heads)
# convert the total_number_of_heads list to a numpy array
model_heads_arr = np.array(total_number_of_heads)  

Output?

import matplotlib.pyplot as plt

# plot the distribution of the number of heads from the simulation
plt.figure(figsize=(4,4))
plt.hist(model_heads_arr, edgecolor='black', density=True)
plt.title(" Number of Heads from Simulation of a Fair Coin")        
plt.xlabel("Number of Heads")
plt.ylabel("Density")
plt.grid()
plt.show()

Output?

Obtain the Observed Data

To truly test a real coin, we would take the coin and toss it 500 times, recording each result.
But for this exercise we'll simulate the coin toss and pretend this is our actual coin toss results with the real coin.

# toss the coin 500 times
actual_results = coin_toss_simulation(500)
# count the number of heads and tails
observed_heads = np.sum(actual_results == 0)
observed_tails = np.sum(actual_results == 1)
print("Number of heads:", observed_heads)
print("Number of tails:", observed_tails)

Output?

Number of heads: 261
Number of tails: 239

# plotting the number of heads in the distribution of the number of heads from the simulation
plt.figure(figsize=(4,4))   
plt.hist(model_heads_arr, edgecolor='black', density=True)
plt.title("Number of Heads from Simulation of a Fair Coin")        
plt.xlabel("Number of Heads")
plt.ylabel("Density")
# add a red circle for the observed number of heads
plt.scatter(observed_heads, 0, color='red', s=100)  
plt.grid()
plt.show()

Output?

Compare Data and Calculate p-value

We can also measure and compare the observed data and the model's data. To do this, we need to identify what the test statistic is. Recall from Chapter 8 that a statistic is a measurement of the data that summarizes an aspect of the data, such as the mean height of people in the dataset. In hypothesis testing the test statistic measures of how far the data is from what we expect in the null hypothesis.

The null hypothesis says that there's no influence that makes the observed data different from the model's data. This means the observed data (from tossing the tested coin) should be similar to the model's data (from tossing the fair coin). And the model's data from tossing the fair coin says that out of 500 coin tosses, we expect 250 will be heads.

Given that 250 is what we expect in the null hypothesis, the test statistic is how far the number of heads is from 250. This means we find the difference between the number of heads and 250.

Comparing the data involves 3 steps:

• find the difference between the observed number of heads and 250. This is the observed distance.
• find the difference between each of the 10,000 simulated numbers of heads and 250. This is an array of simulated distances.
• then we find what proportion of the simulated distances is larger (or farther from 250) than the observed distance.

If the proportion is large (>= 0.05), then the observed number is well within the range of what the null hypothesis says, since the simulated result is the data of the null hypothesis.
If the percentage is small (< 0.05), then the observed number is outside the range of what the null hypothesis says.

This proportion is called the p-value. If the p-value is large, then we can reject the relative hypothesis.
And if the p-value is small, it is statistically significant and we can reject the null hypothesis.

observed_distance = abs(observed_heads - 250)
simulated_distances = np.abs(model_heads_arr - 250)
p_value = np.sum(simulated_distances >= observed_distance)/10000 
# or:
p_value = np.mean(simulated_distances >= observed_distance)
print("p-value:", p_value)

Output?

p-value: 0.3487

# show where the observed_distance is in a distribution of the simulated_distances

plt.figure(figsize=(4,4))
plt.hist(simulated_distances, edgecolor='black', density=True)
plt.xlabel('Distance from Expected Value (250)')
plt.ylabel('Frequency')
plt.title('Distribution of Simulated Distances from Expected Value')
# add a red circle for the observed distance
plt.scatter(observed_distance, 0, color='red', s=100)
plt.grid()
plt.show()

Output?