25  Limit Theorems

Sample mean

Definition 25.1 (Sample mean) Let X_{1}, \dots , X_{n} be a sequence of i.i.d random variables with mean \mu and variance \sigma^{2}. Then the sample mean \bar{X}_{n} is defined as

\bar{X}_{n} = \frac{ 1 }{ n } \sum_{i = 1}^{n} X_{i}.

Since the sample mean is a function of random variables and it can be different for different samples of n i.i.d random variables, it is also a random variable. This means that the sample mean itself has the expectation and variance, which can be calculated as follows:

\mathbb{E}_{\bar{X}_{n}} [\bar{X}_{n}] = \mathbb{E}_{X_{1}, \dots, X_{n}} \left[ \frac{ 1 }{ n } \sum_{i = 1}^{n} X_{i} \right] = \frac{ 1 }{ n } \mathbb{E}_{X_{i}} [X_{i}] = \frac{ 1 }{ n } n \mu = \mu,

\mathrm{Var} [\bar{X}_{n}] = \mathrm{Var} \left[ \frac{ 1 }{ n } \sum_{i = 1}^{n} X_{i} \right] = \frac{ 1 }{ n^{2} } \sum_{i = 1}^{n} \mathrm{Var} [X_{i}] = \frac{ 1 }{ n^{2} } n \sigma^{2} = \frac{ \sigma^{2} }{ n }.

Law of large numbers (LLN)

There are two versions laws of large numbers, both of which state that the the sample mean of n i.i.d random variables converges to their mean \mu, that is, as n get larger, the sample mean is getting closer to \mu.

Theorem 25.1 (Weak law of large number (WLLN)) Let \bar{X}_{n} be the sample mean of n i.i.d random variables X_{1}, \dots , X_{n} with mean \mu. Then \bar{X}_{n} converges in probability to \mu

\lim_{n \to \infty} \mathbb{P} (\lvert \bar{X}_{n} - \mu \rvert > \epsilon) = 0, \quad \epsilon > 0.

Let X_{1}, \dots, X_{n} be i.i.d random variables with finite mean \mu and finite variance \sigma^{2}. Then for any a > 0

\mathbb{P}_{X_{1}, \dots, X_{n}} \left( \left\lvert \frac{\sum_{i = 1}^{n} x_{i}}{n} - \mu \right\rvert \geq \epsilon \right) \leq \frac{\sigma^{2}}{n \epsilon^{2}}

Applying the Chebyshev’s inequality Theorem 23.2 over multiple random variables, we get the following for any t > 0,

\begin{aligned} \mathbb{P} \left( \left\lvert \sum_{i = 1}^{n} X_{i} - \sum_{i = 1}^{n} \mu_{i} \right\rvert \geq t \right) & \leq \frac{ \sum_{i = 1}^{n} \sigma_{i}^{2} }{ t^{2} } \\ \mathbb{P} \left( \left\lvert \sum_{i = 1}^{n} X_{i} - n \mu \right\rvert \geq t \right) & \leq \frac{ n \sigma^{2} }{ t^{2} }. \\ \end{aligned}

Setting t = n \epsilon,

\begin{aligned} \mathbb{P} \left( \left\lvert \sum_{i = 1}^{n} X_{i} - n \mu \right\rvert \geq n \epsilon \right) & \leq \frac{n \sigma^{2}}{n^{2} \epsilon^{2}} \\ \mathbb{P} \left( \left\lvert \frac{ \sum_{i = 1}^{n} X_{i} }{ n } - \mu \right\rvert \geq \epsilon \right) & \leq \frac{\sigma^{2}}{n \epsilon^{2}}. \\ \mathbb{P} \left( \lvert \bar{X}_{n} - \mu \rvert \geq \epsilon \right) & \leq \frac{\sigma^{2}}{n \epsilon^{2}}. \end{aligned}

We can get WLLN by taking the limit n \to \infty

\lim_{n \to \infty} \mathbb{P} \left( \lvert \bar{X}_{n} - \mu \rvert \geq \epsilon \right) = 0.

Theorem 25.2 (Strong law of large number (SLLN)) Let \bar{X}_{n} be the sample mean of n i.i.d random variables X_{1}, \dots , X_{n} with mean \mu. Then \bar{X}_{n} converges almost surely to \mu

\mathbb{P} (\lim_{n \to \infty} \bar{X}_{n} = \mu) = 1.

TODO

WLLN is form of convergence in probability, while SLLN is form of almost sure convergence. Therefore, SLLN is a stronger version than the WLLN.

Central limit theorems

Theorem 25.3 (Central limit theorem (CLT)) Let \bar{X}_{n} be the sample mean of n i.i.d random variables X_{1}, \dots , X_{n} with mean \mu and variance \sigma^{2}. If n goes to infinite, then \bar{X}_{n} follows a Gaussian distribution with mean \mu and \frac{ \sigma^{2} }{ n },

\bar{X}_{n} \sim \mathcal{N} \left( \mu, \frac{ \sigma^{2} }{ n } \right).

TODO

Although CLT is a form of convergence in distribution, which is known to be a weaker version of convergence than convergence in probability and almost sure convergence, it doesn’t mean that CLT is a weaker version of SLLN or WLLN.

Confidence intervals

The sample mean \bar{X}_{n} gives us a single number as our best guess for \mu, but it doesn’t tell us how much we should trust that guess. A confidence interval addresses this by giving a range of values that is likely to contain \mu, instead of just a single point.

Definition 25.2 (Confidence interval) Let X_{1}, \dots , X_{n} be a sequence of random variables with mean \mu. A confidence interval for \mu with confidence level 1 - \alpha, \alpha \in (0, 1), is an interval [L, U], computed from X_{1}, \dots, X_{n}, that satisfies

\mathbb{P} (L \leq \mu \leq U) = 1 - \alpha.

Remark. Since \mu is a fixed constant, all the randomness in \mathbb{P} (L \leq \mu \leq U) comes from L and U, which are functions of the random variables X_{1}, \dots, X_{n} (Definition 19.1) and are therefore random variables themselves, even before we observe any data. So Definition 25.2 is a statement about how likely the interval is to cover \mu, not a statement about \mu itself.

Once we observe the realizations x_{1}, \dots, x_{n} and compute the realized interval [l, u], this interval either contains \mu or it doesn’t, so it no longer makes sense to say “\mu falls in [l, u] with probability 1 - \alpha.” The correct interpretation is that if we repeat the sampling process many times and compute [L, U] each time, about (1 - \alpha) \times 100\% of the resulting intervals will contain \mu.

Confidence interval for the mean

By Theorem 25.3, as n \to \infty, the sample mean follows a Gaussian distribution

\bar{X}_{n} \sim \mathcal{N} \left( \mu, \frac{ \sigma^{2} }{ n } \right),

which we use as an approximation when n is large but finite. Standardizing \bar{X}_{n}, that is, subtracting its mean and dividing by its standard deviation, gives us a random variable Z that follows the standard normal distribution

Z = \frac{ \bar{X}_{n} - \mu }{ \sigma / \sqrt{n} } \sim \mathcal{N} (0, 1).

Remark. Z has mean 0 and variance 1 by Corollary 20.1 and Corollary 20.5

\mathbb{E}_{Z} [Z] = \frac{ \mathbb{E}_{\bar{X}_{n}} [\bar{X}_{n}] - \mu }{ \sigma / \sqrt{n} } = \frac{ \mu - \mu }{ \sigma / \sqrt{n} } = 0, \quad \mathrm{Var} [Z] = \frac{ \mathrm{Var} [\bar{X}_{n}] }{ (\sigma / \sqrt{n})^{2} } = \frac{ \sigma^{2} / n }{ \sigma^{2} / n } = 1,

and by ?cor-standardization, standardizing a Gaussian random variable this way always gives the standard normal distribution, so Z \sim \mathcal{N} (0, 1).

By ?def-standard-normal-quantile, the upper \alpha / 2 quantile z_{\alpha / 2} of the standard normal satisfies

\mathbb{P} \left( -z_{\alpha / 2} \leq Z \leq z_{\alpha / 2} \right) = 1 - \alpha.

Corollary 25.1 (Confidence interval for the mean) Let \bar{X}_{n} be the sample mean of n i.i.d random variables X_{1}, \dots , X_{n} with mean \mu and variance \sigma^{2}. Then a confidence interval for \mu with confidence level 1 - \alpha is

\left[ \bar{X}_{n} - z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} }, \quad \bar{X}_{n} + z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} } \right].

By the definition of z_{\alpha / 2},

\begin{aligned} 1 - \alpha & = \mathbb{P} \left( -z_{\alpha / 2} \leq Z \leq z_{\alpha / 2} \right) \\ & = \mathbb{P} \left( -z_{\alpha / 2} \leq \frac{ \bar{X}_{n} - \mu }{ \sigma / \sqrt{n} } \leq z_{\alpha / 2} \right) \\ & = \mathbb{P} \left( -z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} } \leq \bar{X}_{n} - \mu \leq z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} } \right) & [\text{multiplying by } \frac{ \sigma }{ \sqrt{n} }] \\ & = \mathbb{P} \left( \bar{X}_{n} - z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} } \leq \mu \leq \bar{X}_{n} + z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} } \right) & [\text{rearranging}]. \end{aligned}

Comparing this to Definition 25.2 with L = \bar{X}_{n} - z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} } and U = \bar{X}_{n} + z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} } proves the corollary.

Remark. A few things to note about the confidence interval for the mean.

  • The term z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} } is often called the margin of error, and the confidence interval is simply the sample mean plus or minus the margin of error.

  • The width of the interval, 2 z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} }, shrinks as n grows, since \mathrm{Var} [\bar{X}_{n}] = \frac{ \sigma^{2} }{ n } (from Definition 25.1) gets smaller. This agrees with the law of large numbers: with more samples, \bar{X}_{n} concentrates more tightly around \mu, so a narrower interval is enough to capture it with the same confidence level.

  • A higher confidence level 1 - \alpha requires a larger z_{\alpha / 2}, and therefore a wider interval, since making it more likely for the interval to cover \mu requires the interval to be wider. For example, z_{0.025} \approx 1.96 for a 95\% confidence interval, while z_{0.005} \approx 2.576 for a 99\% confidence interval.

  • In practice, \sigma is usually unknown as well and needs to be estimated from the sample, which introduces additional uncertainty that this formula doesn’t account for.