6.3: Probability Theory and Key Distributions

Probability Theory and Key Distributions

While descriptive statistics summarize what we observed, probability theory lets us model and reason about what we might observe under certain assumptions. Probability provides the mathematical foundation for inference: it describes the behavior of random variables, i.e. numerical outcomes of random processes. In data analytics, probability models are used to anticipate variability and to calculate how likely or unlikely certain results are, assuming a given model.

Basics of Probability

In simple terms, probability is the chance of an event occurring, expressed as a number between 0 and 1 (or 0% to 100%). A probability of 0 means the event is impossible; 1 means it is certain. For example, the probability of getting a 6 in a fair die roll is 1/6 (≈0.167 or 16.7%). When we talk about random events (like flipping a coin, tomorrow’s rainfall, or a customer arriving in the next minute), probability theory gives us tools to quantify uncertainty.

Key concepts in probability include: - Random experiment: a process leading to two or more possible outcomes, where there’s uncertainty which outcome will occur on any given trial (e.g., rolling a die). - Sample space: the set of all possible outcomes of an experiment (for a die roll, {1,2,3,4,5,6}). - Event: a subset of outcomes of interest (e.g., the event “roll an even number” = {2,4,6}). - Probability distribution: a function that assigns probabilities to outcomes or events. For discrete outcomes, we have a probability mass function (PMF) that gives $P(X = x)$ for each possible value $x$. For continuous outcomes, a probability density function (PDF) gives a density and probabilities are areas under the curve.

Probabilities obey certain rules (the axioms of probability). For instance, the probability of all possible outcomes sums to 1. The probability of either event A or event B happening (if mutually exclusive) is $P(A)+P(B)$. The probability of two independent events A and B both happening is $P(A)\times P(B)$. Understanding these rules and definitions is crucial for working with statistical models.

Probability Distributions

A probability distribution specifies how probabilities are distributed over the values of a random variable. There are many named distributions, each useful for different scenarios. We will introduce a few of the most important ones in analytics:

• Normal Distribution (Gaussian) – The normal distribution is a continuous distribution famous for its bell-shaped density curve. It is symmetric around the mean, with data near the mean more frequent in occurrence than data far from the mean [3]. Formally, a normal distribution $N(\mu, \sigma^2)$ has mean $\mu$ (also the median and mode, since it’s symmetric) and variance $\sigma^2$. Its PDF is given by:

$$f(x) = \frac{1}{\sigma\sqrt{2\pi}} \exp!\Big(-\frac{(x-\mu)^2}{2\sigma^2}\Big),$$

which produces the bell curve shape. The normal is often called Gaussian after Carl F. Gauss. Many natural phenomena are approximately normal (heights, measurement errors, IQ scores, etc.) due to the Central Limit Theorem (CLT): roughly speaking, the sum (or average) of many small independent effects tends to be normally distributed. The normal distribution is immensely important in statistics; for example, many statistical inference techniques assume normally-distributed errors or sampling distributions.

Figure \(\PageIndex{1}\):Measures of Central Tendency in Skewed vs. Symmetric Distributions. Three distribution curves illustrate how mean, median, and mode align in a symmetric distribution but diverge in left-skewed and right-skewed distributions. (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{1}\)

The standard normal (mean 0, $\sigma=1$) is a baseline; any $N(\mu,\sigma^2)$ can be converted to standard form via a Z-score transformation: $Z = (X-\mu)/\sigma$, which has $N(0,1)$. Normal distribution tables or functions (like scipy.stats.norm.cdf in Python) give areas under the curve (i.e. probabilities) for standard normals, which can be translated to any normal via Z-scores.

In practice, the normal distribution is often used as an approximation or assumption. For example, by CLT, the sample mean of sufficiently large samples is approximately normal around the true mean. Measurement errors are frequently assumed normal. When data appear roughly symmetric and bell-shaped in a histogram, analysts might fit a normal model to it. However, not all symmetric distributions are normal, and it’s important to check normality assumptions (e.g. via Q-Q plots). If data are significantly skewed or have heavy tails, other distributions or non-parametric methods might be more appropriate.

• Binomial Distribution – The binomial distribution is a discrete distribution that models the number of successes in a fixed number of independent Bernoulli trials (trials with two possible outcomes: success or failure) [5]. It has two parameters: n (number of trials) and p (probability of success on each trial). If $X \sim \text{Binomial}(n, p)$, then $X$ can take integer values $0,1,2,\dots,n$, and the probability of getting exactly $k$ successes is given by the binomial formula:

$$P(X = k) = \binom{n}{k} p^k (1-p)^{\,n-k},$$

for $k=0,1,\ldots,n$. Here $\binom{n}{k} = \frac{n!}{k!(n-k)!}$ is the number of ways to choose which $k$ trials out of $n$ are successes. This formula comes from the fact that any specific sequence of $k$ successes and $n-k$ failures has probability $p^k(1-p)^{n-k}$, and there are $\binom{n}{k}$ such sequences. The expected value (mean) of $X$ is $np$, and the variance is $np(1-p)$.

The binomial distribution is appropriate when you have a fixed number of identical, independent trials and want the distribution of the count of successes. Classic examples: number of heads in 10 coin flips (with p=0.5 for heads), number of defective items in a batch of 50 (if each item has, say, p=0.02 chance of defect), number of customers who buy versus just browse out of 100 visitors (with some probability of purchase).

The binomial distribution is the workhorse for binary outcome scenarios. It also underpins the binomial test in statistics, which checks if an observed number of successes in $n$ trials is consistent with a hypothesized probability $p$. For instance, if you observe 8 heads in 10 flips, you might use a binomial model to ask if the coin could be fair ($p=0.5$) or if 8/10 is unusually high (we will revisit hypothesis testing soon) [10]. The binomial model also forms the basis for the popular Bernoulli (0/1) modeling in fields like A/B testing and logistic regression.

Figure \(\PageIndex{2}\): Binomial Distribution Bar Chart (n=10, p=0.5). A bar chart displaying the probability mass function of a Binomial(10, 0.5) distribution, showing the symmetric distribution of probabilities across 0 to 10 successes. (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{2}\)

 

Figure \(\PageIndex{3}\): Binomial Distribution Bar Chart (n=10, p=0.5). A bar chart displaying the probability mass function of a Binomial(10, 0.5) distribution, showing the symmetric distribution of probabilities across 0 to 10 successes. (CC BY 4.0; Felix Amoruwa) Access to a detailed description of Figure \(\PageIndex{3}\)

Poisson Distribution – The Poisson is a discrete distribution often used for modeling counts of events that occur randomly over a continuous domain (time, area, etc.), especially when the events are rare or at a low rate [7]. A Poisson random variable $X \sim \text{Poisson}(\lambda)$ takes values $0,1,2,\ldots$ with probability.

$$P(X = k) = \frac{\lambda^k e^{-\lambda}}{k!}, \quad k = 0,1,2,\ldots$$

The single parameter $\lambda$ is both the expected value and the variance of the distribution. Intuitively, $\lambda$ represents the average rate (mean count) of events in the given interval. The Poisson distribution arises theoretically as a limit of the binomial distribution for large $n$ and small $p$ (with $np=\lambda$ fixed). It is often appropriate when events occur independently and at a roughly constant average rate.

Examples of Poisson processes: the number of customer arrivals at a store in an hour, number of website hits per minute, number of defects in a length of fabric, or counts of rare events (like mutations on a DNA strand, or calls to a call center). The key assumption is that events occur independently and not in clumps, with a steady rate. If a call center receives on average $\lambda=3$ calls per minute, the probability of receiving exactly $k$ calls in a given minute would follow a Poisson($3$) distribution. For instance, $P(X=0) = e^{-3}3^0/0! = e^{-3}\approx 0.05$ (5% chance no calls in that minute), $P(X=3) = e^{-3}3^3/3! \approx 0.224$ (about 22.4% chance of exactly 3 calls), and so on.

The Poisson is often used in count data models. One hallmark is that if events are Poisson-distributed, the variance equals the mean $\lambda$. If data show that variance is higher than the mean (over-dispersion), other models (like the negative binomial) might be more appropriate. Still, the Poisson remains a convenient model for counts and is used widely in queuing theory, reliability engineering, and anywhere one needs to model random counts of events in intervals. 

• Uniform Distribution – The uniform distribution is the simplest distribution, where all outcomes in a certain range are equally likely. It comes in two flavors:
• Discrete uniform: e.g. rolling a fair die has a uniform distribution over {1,2,…,6}, each outcome with probability 1/6.
• Continuous uniform: e.g. if a random number is picked uniformly between 0 and 1, any sub-interval of equal length has the same probability of containing the outcome. A continuous uniform on interval $[a,b]$ has constant density $f(x) = 1/(b-a)$ for $a \le x \le b$.

The uniform distribution is often described as “rectangular” because of its flat shape on a plot. For a continuous uniform $[a,b]$, the mean is $(a+b)/2and variance is $(b-a)^2/12. The uniform is useful as a benchmark of randomness – for example, many random number generators are designed to produce nearly i.i.d. (independent identically distributed) uniform(0,1) numbers. Also, when we have no information favoring any outcome over another in an interval, a uniform model might be assumed as a non-informative prior.

From a data analytics perspective, uniform distribution examples include: modeling a random guess on a multiple-choice question (each option equally likely), or using the uniform as a stepping stone to generate other distributions (e.g. the inversion method for simulation uses uniform(0,1) draws to produce values from other distributions). Uniform is also the null distribution for certain tests; interestingly, if a null hypothesis is true, p-values are uniformly distributed between 0 and 1 (assuming continuous test statistics) [8]. This property is used in statistical diagnostics.

In summary, these four distributions – normal, binomial, Poisson, uniform – are foundational: - The normal distribution often describes continuous data influenced by many small random effects and is the basis of many parametric statistical tests. - The binomial distribution handles discrete counts of successes in fixed trials, common in yes/no type experiments. - The Poisson distribution handles counts of events over continuous space or time, especially for rare events. - The uniform distribution represents “complete randomness” with no bias to any outcome.

Each distribution has its own parameters and characteristics (shape, mean, variance) and is appropriate in different contexts. Choosing the right probability model for your data (or for the stochastic process you’re analyzing) is an important step in any analysis. Often, model selection involves looking at the data’s histogram and comparing to theoretical shapes, or performing goodness-of-fit tests. Statistical software and libraries (like SciPy) provide functions for these distributions – both to compute probabilities (PMFs, PDFs, cumulative probabilities) and to draw random samples for simulation. For example, in Python:

import numpy as np
from scipy import stats
# Draw 5 random samples from a normal distribution with mean 10, std 2:
samples = np.random.normal(loc=10, scale=2, size=5) 
# Probability of getting exactly 4 successes in 10 trials with p=0.3:
prob_4 = stats.binom.pmf(4, n=10, p=0.3)
# Probability of 2 or fewer events when lambda=5 (Poisson CDF):
prob_le2 = stats.poisson.cdf(2, mu=5)

These tools allow analysts to calculate odds of scenarios and simulate outcomes under assumed models, which is invaluable for risk assessment and planning.

Before moving on, a special mention: the Central Limit Theorem (CLT) was hinted at. It states that the sampling distribution of the sample mean (or sum) of a large number of independent, identically distributed variables tends toward a normal distribution, regardless of the original distribution of the variables (given finite variance). This theorem justifies why the normal is so pervasive and is a cornerstone of inferential statistics, which we discuss next.