6.4: Inferential Statistics - From Samples to Populations

Overview

Descriptive statistics and probability distributions set the stage for inferential statistics, which involves drawing conclusions about a population based on a sample of data. Since analyzing an entire population is often impractical (too expensive or time-consuming), we rely on samples. Inferential methods include estimation (e.g., confidence intervals) and hypothesis testing, which together allow us to quantify uncertainty and make data-driven decisions with a known level of confidence or risk of error.

Populations, Samples, and Sampling

A population is the entire group of interest (e.g., all customers of a company, all residents of a country, all widgets produced in a factory). A sample is a subset of the population that is actually observed or measured. The goal is often to use sample statistics (like the sample mean $\bar{x}$ or sample proportion $\hat{p}$) as estimates of the true population parameters (like population mean $\mu$ or population proportion $p$).

Key points: - We typically assume the sample is randomly selected from the population. Random sampling helps ensure the sample is representative, so that statistical inference is valid. A simple random sample gives each member of the population an equal chance to be chosen. - Every sample will vary due to chance (sampling error). If you took a different random sample, you’d get different values. This variability is the basis of inferential statistics – we can model it and account for it. - A sampling distribution is the probability distribution of a statistic (like the mean) over all possible samples. For example, the sampling distribution of the sample mean $\bar{X}$ has mean $E(\bar{X})=\mu$ and standard deviation $\sigma_{\bar{X}} = \sigma/\sqrt{n}$ (where $\sigma$ is the population SD and $n$ the sample size). $\sigma_{\bar{X}}$ is known as the standard error of the mean. The CLT says that for large $n$, this distribution is approximately normal. - Bias and sampling errors: A statistic is an unbiased estimator of a parameter if its expected value equals the true parameter. Using $n-1$ in sample variance formula, for instance, makes it an unbiased estimator of $\sigma^2$. A sample is biased if the sampling method systematically over- or under-represents certain groups (e.g., surveying only daytime internet users misses people who work daytime). Avoiding sampling bias is crucial – “garbage in, garbage out” applies to data.

In practice, one must be mindful of how the sample is collected. Inferential formulas assume random sampling. If data come from an observational study with some selection bias, the inferences might be invalid. Assuming we have a good sample, we proceed to either estimate population parameters or test hypotheses.

Point Estimates and Confidence Intervals

A point estimate is a single number that is our best guess for a population parameter. For example, $\bar{x}$ is a point estimate of $\mu$, and the sample proportion $\hat{p}=X/n$ (with X successes in n trials) is a point estimate of the population proportion $p$. However, a point estimate alone does not quantify uncertainty. That’s where confidence intervals (CIs) come in.

A confidence interval provides a range of plausible values for the population parameter, along with a confidence level that quantifies how confident we are that the range contains the true value [9]. Formally, a confidence interval is a range of values that is likely to contain a population parameter with a certain level of confidence. Typically, we talk about 90%, 95%, or 99% confidence intervals. A 95% CI means that if we repeated the sampling procedure many times, about 95% of the intervals calculated would contain the true parameter.

The general form of a confidence interval is:

$$\text{Estimate} \ \pm\ (\text{critical value}) \times (\text{standard error}).$$

The critical value comes from the sampling distribution for the chosen confidence level, and the standard error is the estimated standard deviation of the statistic.

For example, a confidence interval for a population mean (assuming we know the population standard deviation $\sigma$ or large $n$ so that we use a z-critical value) is:

$$\bar{x} \ \pm\ z_{\alpha/2}\ \frac{\sigma}{\sqrt{n}},$$

where $z_{\alpha/2}$ is the z-score cutting off $\alpha/2$ in the upper tail (for 95% confidence, $\alpha=0.05$ and $z_{0.025}\approx1.96$). If $\sigma$ is unknown and $n$ is not very large, we use the t-distribution with $n-1$ degrees of freedom instead of z. In that case the formula is similar but with a t-critical value and $s$ (sample standard deviation) in place of $\sigma$.

Interpretation

Suppose we compute a 95% CI for the mean weight of turtles as $[293.9, 306.1]$ (in whatever units). We would interpret this as: "We are 95% confident that the true mean weight of the turtles lies between 293.9 and 306.1." This does not mean there is a 95% probability the true mean is in that range (the true mean is fixed, not random). Rather, it means that our method of constructing intervals will capture the true mean 95% of the time in repeated samples. In one given sample, either the interval covers $\mu$ or it doesn’t – but we don’t know. The confidence level quantifies our long-run success rate.

Figure \(\PageIndex{1}\): Confidence Interval Construction and Interpretation. A diagram showing how repeated sampling produces different 95% confidence intervals, most of which capture the true population parameter, illustrating the frequentist interpretation of confidence. (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{1}\)

CIs exist for many parameters

Mean, difference in means, proportions, difference in proportions, regression coefficients, etc. For example, a CI for a population proportion $p$ can be computed with the Wilson or normal approximation method: $\hat{p} \pm z \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}$ (for large n). There are also CIs for more complex statistics like the difference between two means (which uses a t value and pooled or unpooled standard error depending on assumptions).

Confidence intervals are extremely useful in practice because they show the precision of our estimates. A narrow 95% CI indicates a precise estimate (likely from a large sample or low variability data), whereas a wide CI indicates more uncertainty. For instance, if one drug has a 95% CI for mean effect [1.0, 3.0] and another drug’s CI is [0.5, 8.0], we have much more precise knowledge about the first drug’s effect. Overlapping CIs can indicate that differences are not statistically significant, though one should be cautious in directly inferring significance from CIs unless they are constructed for differences explicitly.

Hypothesis Testing and p-Values

While confidence intervals focus on estimation, hypothesis testing addresses decision-making: testing specific claims about population parameters. In hypothesis testing, we set up: - Null hypothesis ($H_0$): A neutral or status-quo assumption (e.g., no effect, no difference). This is the hypothesis we assume true initially and test against. Often $H_0$ is a statement like “the population mean equals 0” or “the two groups have the same mean” or “proportion $p=0.5$.” - Alternative hypothesis ($H_a$): The claim we seek evidence for (e.g., there is an effect, there is a difference, the coin is biased). This can be one-sided (e.g. $p>0.5$) or two-sided ($p \neq 0.5$) depending on the question.

We collect data and calculate a test statistic, which is designed to measure how far the sample deviates from $H_0$ in the direction of $H_a$. Examples: a t-statistic for testing a mean, a Z-statistic for proportions or large-sample means, a chi-square for categorical counts, etc. We then find the p-value of the test result.

The p-value is defined as the probability of obtaining results as extreme as (or more extreme than) the observed results, assuming that the null hypothesis is true. In other words, it answers: If $H_0$ were true, how surprising would our data be? A small p-value indicates the data would be very unusual under $H_0$, thus providing evidence against $H_0$.

By convention, we choose a significance level $\alpha$ (often 0.05). If p-value $\le \alpha$, we reject $H_0$ and conclude the result is statistically significant (in favor of $H_a$). If p-value $> \alpha$, we fail to reject $H_0$ (we do not have sufficient evidence to support $H_a$). Note: “fail to reject” is not the same as proving $H_0$ true – it just means not enough evidence to say otherwise.

For example, imagine we have a fair coin null hypothesis $H_0$: $p=0.5$ (probability of heads). We flip the coin 100 times and observe 60 heads. Is this coin biased towards heads? Our test statistic could be the number of heads $X=60$ itself. Under $H_0$, $X \sim \text{Binomial}(100,0.5)$. We can compute the p-value as the probability of getting 60 or more heads out of 100 if $p=0.5$ (this is a one-sided test for bias towards heads).

Using Python/SciPy

from scipy import stats
p_value = stats.binom_test(60, n=100, p=0.5, alternative='greater')
print("p-value:", p_value)

This yields p-value: 0.028 (approximately). That means there is about a 2.8% chance of getting 60 or more heads if the coin were fair. If our $\alpha$ was 0.05, the p-value 0.028 is below $\alpha$, so we reject $H_0$ and conclude the coin is likely biased towards heads. In plain language: if the coin were truly fair, the chance of seeing as extreme a result as 60 heads in 100 flips is only ~2.8%, which is quite unlikely, so we suspect the coin is not fair.

Another example: a clinical trial might test $H_0$: “new drug is no better than placebo” vs $H_a$: “new drug is better.” We collect data on patient improvements. If the p-value of the observed improvement is, say, 0.01 under $H_0$, we reject $H_0$ (at $\alpha=0.05$) and conclude the drug has a statistically significant effect.

It is important to note that the p-value is not the probability that $H_0$ is true; it’s computed assuming $H_0$ is true. Nor does a non-significant result prove $H_0$ – it may simply mean the data were inconclusive.

Significance level and errors

By choosing $\alpha=0.05$, we are allowing a 5% chance of a Type I error (false positive, rejecting a true $H_0$). A Type II error is failing to reject a false $H_0$ (false negative), and its probability is denoted by $\beta$ (the power = $1-\beta$ is the probability of correctly detecting an effect when there is one). Typically, one designs experiments to control $\alpha$ and maximize power (often aiming for power 0.8 or 0.9). In some contexts, more stringent $\alpha$ are used (e.g. 0.01) if false positives are very problematic.

Hypothesis tests appear throughout analytics

A/B testing in business is essentially hypothesis testing (is conversion rate different under layout A vs B?), quality control uses tests (is defect rate <= some target?), social science research uses t-tests/ANOVAs (are group means different?), etc. Each scenario has its own appropriate test (z, t, chi-square, F, etc.), but the core logic is similar: 1. Define null and alternative. 2. Compute test statistic from data. 3. Find p-value given null distribution. 4. Decide to reject or not reject null based on p vs $\alpha$.

Figure \(\PageIndex{2}\): Hypothesis Testing Decision Framework. A flowchart illustrating the step-by-step process of hypothesis testing, from formulating hypotheses through computing a test statistic and p-value to reaching a conclusion. (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{2}\) 

 

Figure \(\PageIndex{3}\): Type I and Type II Errors in Hypothesis Testing. A two-by-two matrix showing the four possible outcomes of a hypothesis test, distinguishing correct decisions from Type I errors (false positives) and Type II errors (false negatives). (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{3}\)

Modern practice often complements hypothesis tests with confidence intervals and effect size estimates, to ensure that statistically significant results are also practically significant and to provide more information than a binary reject/accept decision.

Example – t-test and Confidence Interval for a Mean

To tie together these concepts, consider a concrete example: We want to see if a new tutoring program affects test scores. Let’s say in the general population, test scores are normally distributed with unknown mean. We collect a sample of 30 students, and their average score is $\bar{x}=78$ with sample standard deviation $s=10$. Suppose without tutoring the average was expected to be 75. We test $H_0: \mu = 75$ vs $H_a: \mu \neq 75$ (two-tailed, since maybe the program could help or hurt).

We calculate a t-statistic

$$ t = \frac{\bar{x}-75}{s/\sqrt{n}} = \frac{78-75}{10/\sqrt{30}} = \frac{3}{10/5.477} \approx \frac{3}{1.826} \approx 1.64.$$

With df = 29, a t of 1.64 (two-tailed) gives a p-value around 0.11 (11%). If $\alpha=0.05$, this is not significant (p > 0.05). We fail to reject $H_0$; we don’t have strong evidence that the program changed scores. Perhaps the sample was too small or the effect is just modest.

We can also compute a 95% confidence interval for $\mu$:

$$78 \pm t_{0.025,29} \times \frac{10}{\sqrt{30}}.$$

For df=29, $t_{0.025}\approx2.045$. So CI = $78 \pm 2.045*(1.826) = 78 \pm 3.73$, i.e. [74.27, 81.73]. This interval includes 75, which is consistent with our failure to reject 75. It suggests the true mean could plausibly be as low as ~74.3 or as high as ~81.7. The interval is somewhat wide; a larger sample would narrow it. If we wanted more confidence (99%), the interval would be even wider.

This example demonstrates how one would perform a test and CI. Typically, one would use software rather than manual calculation. For instance, using Python’s SciPy or statsmodels:

import numpy as np
from scipy import stats
scores = np.array([...])  # sample data
# One-sample t-test against mean = 75:
t_stat, p_val = stats.ttest_1samp(scores, popmean=75)
# Confidence interval for the mean:
mean = scores.mean()
sem = stats.sem(scores)  # standard error of the mean
ci95 = stats.t.interval(0.95, df=len(scores)-1, loc=mean, scale=sem)

This would output the t-statistic, p-value, and the 95% CI. Stats models also provides more detailed outputs.

Correlation, Regression, and Causality

Correlation and regression are tools for examining relationships between two or more variables [2]. They overlap with inferential statistics (e.g., testing if a correlation is significantly different from zero, or if a regression slope is nonzero), but we’ll treat them in their own category because they specifically address association and modeling. Finally, we’ll discuss causality – which is the ultimate question in many analyses: does a change in one variable cause a change in another? Establishing causality is tricky and goes beyond statistics alone, requiring careful study design or assumptions.

Correlation

Correlation measures the strength and direction of an association between two quantitative variables. The most common measure is the Pearson correlation coefficient (r), which ranges from –1 to +1. An $r$ of +1 means a perfect positive linear relationship (as one variable increases, the other increases in exact proportion), $r$ of –1 means a perfect negative linear relationship (one increases, the other decreases in exact proportion), and $r=0$ means no linear relationship (though there could be a nonlinear relationship). In formula form,

$$ r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i-\bar{x})^2 \sum (y_i-\bar{y})^2}}, $$

which essentially standardizes the covariance between X and Y by the product of their standard deviations. This formula yields a unitless measure of linear association.

Interpretation

If $r \approx 0.8$, that’s a strong positive correlation (when X is high, Y tends to be high). If $r \approx -0.3$, that’s a moderate negative correlation (when X is high, Y tends to be somewhat low). If $r \approx 0$, it suggests no linear trend (though data could still have a pattern, like a U-shape, which Pearson’s $r$ wouldn’t catch).

It’s important to visualize data with a scatter plot when assessing correlation. Points tightly clustered around a line indicate a high |r|. If points are more diffuse, |r| is lower. Also, outliers can distort correlation: a single outlier can either inflate or deflate $r$. Sometimes data might show a relationship but not linear (e.g. a curved pattern) – Pearson’s $r$ might be near 0 even though a strong relationship exists (just not linear).

One can statistically test if a correlation is significantly different from zero by a t-test ($t = r\sqrt{\frac{n-2}{1-r^2}}$ with df = n-2). More generally, computing a correlation matrix for multiple variables can help find interesting associations to explore further.

Example (Python):

import numpy as np
# Generate an example where y is roughly correlated with x
x = np.arange(0, 10)
y = 2*x + np.random.normal(0, 1, size=10)  # y ~ 2x plus some noise
r_matrix = np.corrcoef(x, y)
print("Correlation coefficient:", r_matrix[0,1])

This will output a correlation close to 1 (since y is largely a linear function of x). If the noise were larger, r would drop.

Regression

Regression analysis is about modeling the relationship between a dependent (response) variable and one or more independent (predictor) variables. The simplest and most common is simple linear regression, which fits a straight line $y = \beta_0 + \beta_1 x + \varepsilon$ to data (x, y), where $\varepsilon$ is an error term. The coefficients $\beta_0$ (intercept) and $\beta_1$ (slope) are estimated from data (often by least squares, which finds the line minimizing the sum of squared residuals).

The slope $\beta_1$ tells us the change in $y$ for a one-unit increase in $x$, on average. For example, if we regress house price on square footage, a slope of 50 means each additional square foot is associated with \$50 increase in price on average. The intercept $\beta_0$ is the model’s predicted $y$ when $x=0$ (sometimes not meaningful if 0 is outside the data range).

Regression outputs also include measures like $R^2$ (coefficient of determination), which is the fraction of variance in $y$ explained by the model. An $R^2$ of 0.85 means 85% of the variability in $y$ is accounted for by the linear relationship with $x$ (the rest is unexplained noise). For simple linear regression, $R^2 = r^2$ (the square of the Pearson correlation between x and y).

Regression extends to multiple regression ($y$ vs multiple $x$ variables) where we have $y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \dots + \beta_p x_p + \varepsilon$. The interpretation of each $\beta_i$ is the effect of $x_i$ on $y$ controlling for other variables. Multiple regression allows adjusting for confounders or combining predictors to improve fit.

Most statistical software (like Python’s statsmodels or R) can fit regressions easily and provide output including coefficient estimates, standard errors, t-statistics, p-values for coefficients (testing if a coefficient is significantly different from 0), overall F-tests for model significance, etc.

Example (Python using statsmodels):

import statsmodels.api as sm
import pandas as pd

# Suppose x and y are our data arrays from above:
df = pd.DataFrame({'x': x, 'y': y})
X = sm.add_constant(df['x'])       # add intercept term
model = sm.OLS(df['y'], X).fit()   # fit ordinary least squares linear model
print(model.params)               # estimated intercept and slope
print("R^2:", model.rsquared)

This will output something like: intercept $\beta_0 \approx$ (some value near 0), slope $\beta_1 \approx 2$ (since we generated y ~ 2*x), and a high $R^2$ (close to 1 with low noise). The model.summary() would show p-values indicating the slope is highly significant, etc.

Regression is a powerful tool not just for description but for prediction. For instance, one could use a multiple regression model to predict a student’s college GPA from their high school GPA, SAT score, and study hours. In business, one might regress sales on advertising spend, price, and seasonality indicators to model how these factors relate to revenue.

However – and this is critical – regression (or correlation) alone does not establish causation. One of the most common pitfalls is to infer that if $x$ and $y$ are correlated or if a regression yields a significant coefficient, then changes in $x$ cause changes in $y$. In truth, there are three possibilities: (1) $x$ causes $y$, (2) $y$ causes $x$, or (3) some other factor causes both (a lurking or confounding variable). Correlation simply indicates an association or pattern.

For example, there is a high correlation between ice cream sales and drowning incidents. Does buying ice cream cause drowning? No – the lurking variable is temperature (in summer both ice cream consumption and swimming (hence drownings) increase). This is a classic example of confounding.

The JMP quote succinctly states: seeing two variables move together doesn’t mean one causes the other – correlation does not imply causation. There could be random coincidence or hidden factors.

Causality

Causality means that changing one factor directly produces a change in another. Establishing causality requires more than observing data; it often requires a properly designed experiment or a causal framework. In statistics, randomized controlled trials (RCTs) are the gold standard for demonstrating causality: by randomly assigning subjects to different treatments, we ensure that (in expectation) there are no systematic differences between groups except the treatment, so any outcome difference can be attributed to the treatment.

In observational studies, causal inference is more challenging. Analysts use techniques like regression with controls, instrumental variables, propensity score matching, or difference-in-differences to try to mimic an experimental comparison and control for confounders. However, these methods rely on assumptions and can never be as definitive as a randomized experiment.

A key concept is that correlations in observational data can be spurious – they might arise from biases or confounding. The earlier example of exercise and skin cancer in the JMP excerpt illustrates this: a naive analysis found a positive correlation between exercise and skin cancer (people who exercised more had more skin cancer). One might hastily think exercise causes skin cancer. But a deeper look revealed that a third variable – amount of sunlight – was driving both (people in sunny climates both exercise outdoors more and get more sun exposure leading to skin cancer). Once this lurking variable is accounted for, exercise itself likely has no causal effect on skin cancer risk.

To establish causation, one strategy is to use well-designed empirical research: for example, do a controlled experiment or find a natural experiment. In the exercise vs heart disease example from JMP, a correlation was found (more exercise associated with less heart disease). To move toward causation, researchers might identify a biological mechanism (exercise increases nitric oxide, improving blood pressure) and perform a randomized trial assigning people to exercise programs to see if heart disease incidence drops. Only with that additional evidence (and consistency across studies) can we claim causality with confidence.

In summary: - Correlation quantifies association. It’s useful for prediction and identifying potential relationships. - Regression provides a predictive equation and the marginal association of each predictor with the outcome, controlling for others. It’s a fundamental tool for both explanation and prediction. - Causality requires satisfying more stringent criteria (like temporality, absence of confounding, and often experimental evidence). Statistics can hint at causality (e.g., through significant associations and by ruling out some alternative explanations), but causal conclusions usually demand more (randomization or strong causal assumptions).

A prudent analyst uses regression and correlation to inform and possibly to predict, but is careful about making causal claims unless the study design supports it. “Correlation does not imply causation” is a mantra to always keep in mind – many common pitfalls in statistical reasoning come from ignoring this fact.