# MATH310 Spring 2026 - Day 26 **Date**: March 18, 2026 **Topic**: Correlation in Random Walks and Generalizations via the Central Limit Theorem **Lecture**: 26 ## Administrative Notes - First class after St. Patrick's Day - About 5 more lectures on random walks - About 5 lectures on recurrence relations (bringing determinism back) - About 10 more work days until end of semester - Friday is a work day, followed by spring break week - Thanks to students for helping identify indexing issues in previous lecture ## Recall: Covariance Between Time-Shifted Random Walks From [[MATH310S26-Day25-Notes|Day 25]], we examined two random walk processes: - $S_{n+k}$: Random walkers at time $n+k$ - $S_n$: Random walkers at time $n$ The key difference: First set has moved $k$ many more steps. We derived: $\text{Cov}(S_{n+k}, S_n) = \sum_{i=0}^{n+k-1} \sum_{j=0}^{n-1} \text{Cov}(X_i, X_j)$ Since the steps $X_i$ are [i.i.d. random variables](https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables): - If $i \neq j$: $\text{Cov}(X_i, X_j) = 0$ (independence) - If $i = j$: $\text{Cov}(X_i, X_i) = V[X]$ (variance) Only terms where indices match survive, giving: $\text{Cov}(S_{n+k}, S_n) = \sum_{m=0}^{n-1} V[X] = n \cdot V[X]$ **Key insights:** 1. Covariance grows linearly with $n$ (shared steps) 2. No dependence on $k$ in final result 3. "Always ends up on step statistics" - walk statistics reduce to step properties ## Normalized Correlation Coefficient ### The Problem of Growth Since covariance grows with $n$, we lose sense of scale. Need normalization. ### Definition of $\rho$ (Correlation Coefficient - [[MATH310S26-Day26-Notes#Geometric Interpretation and Unification of Correlation|Geometric Interpretation and Unification of Correlation]]) To make the quantity unitless (removing length²): $\rho_{n+k,n} = \frac{\text{Cov}(S_{n+k}, S_n)}{\sigma_{S_{n+k}} \cdot \sigma_{S_n}}$ Substituting our results: - Numerator: $n \cdot V[X]$ - $\sigma_{S_{n+k}} = \sqrt{(n+k) \cdot V[X]}$ (from [[MATH310S26-Day24-Notes|Day 24]]) - $\sigma_{S_n} = \sqrt{n \cdot V[X]}$ After simplification: $\rho_{n+k,n} = \frac{n \cdot V[X]}{\sqrt{(n+k) \cdot V[X]} \cdot \sqrt{n \cdot V[X]}} = \frac{\sqrt{n}}{\sqrt{n+k}}$ ### Interpretation **Note 1: Fixed $n$ with $k \gg 1$** - As $k$ increases, $\rho \to 0$ - Process becomes decorrelated - Walkers that continue walking lose correlation with past **Note 2: Fixed $k$ with $n \gg 1$** - $\sqrt{n+k} \approx \sqrt{n}$ - $\rho \to 1$ - Long shared history implies strong correlation This is related to the [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) and normalized covariance, bounded between -1 and 1. ## Connection to Markov Chains The random walk is: - An additive process - Built on i.i.d. steps from population with defined mean and variance - A discrete-time stochastic process ([Markov chain](https://en.wikipedia.org/wiki/Markov_chain)) Properties: - If walk continues far into future, little connection to past - If walkers share large past but few differing steps, they remain similar ## Central Limit Theorem Application ### Statement (from Dr. William Navidi) For random variables $X_1, X_2, ..., X_n$ drawn from population with mean $\mu$ and variance $\sigma^2$: If $S_n = \sum_{i=1}^{n} X_i$ and $n$ is sufficiently large: $S_n \sim N(n\mu, n\sigma^2)$ The distribution becomes asymptotically [normal](https://en.wikipedia.org/wiki/Normal_distribution). **Note**: Dr. Navidi (now retired) was an award-winning statistics professor at Mines, academic great-grandchild of [Richard Courant](https://en.wikipedia.org/wiki/Richard_Courant), who brought applied mathematics to America in the 1930s. ### Generalizing Beyond Simple Random Walk The [Central Limit Theorem](https://en.wikipedia.org/wiki/Central_limit_theorem) tells us: - Take steps from ANY population with well-defined mean and variance - For large $n$, histogram looks normally distributed - As $n$ increases, distribution gets wider - Non-zero expected value causes drift ### Examples with Different Step Distributions #### Uniform Distribution - Steps drawn uniformly from $[A, B]$ - Example: $A = -2$, $B = 5$ - Center of mass at $(A+B)/2 = 1.5$ - Result: Walk drifts right while spreading - Creates "river delta" patterns #### Exponential Distribution - Well-defined first and second moments - Asymptotically approaches normal distribution - Despite initial skewness, CLT applies #### Other Distributions Tested - [Gamma distribution](https://en.wikipedia.org/wiki/Gamma_distribution) - [Weibull distribution](https://en.wikipedia.org/wiki/Weibull_distribution) All showed: - Mean error < 0.5% quickly - Standard deviation error generally < 3% - Good agreement with CLT predictions ### Connection to Noise Theory If noise = accumulation of many small effects: - Individual effects we can't track - Together they affect system (e.g., Styrofoam ball in water) - Long-term accumulation → Normal distribution Therefore: Residuals in models should look normally distributed, as if from accumulated lower-order random processes. ## When CLT Fails: Heavy-Tailed Distributions ### The Cauchy Distribution The [Cauchy distribution](https://en.wikipedia.org/wiki/Cauchy_distribution): $f(x) = \frac{1}{\pi(1 + x^2)}$ Properties: - Power law decay: $\sim 1/x^2$ for large $|x|$ - **No defined mean or variance** - Heavy tails → high probability of extreme values - More outliers than normal distribution ### Random Walk with Cauchy Steps Visualization shows: - Most walkers stay central - Occasional extreme jumps (thousands of units in one step) - Histogram requires very long x-axis - Some walkers "go crazy far" unpredictably ### Real-World Example: Andre the Giant André René Roussimoff: - Height: 7'4" - Weight: 520 pounds Under normal distribution assumptions: - Probability of his height: $10^{-248}$ - Probability of his weight: $10^{-9}$ These probabilities are essentially impossible under normal distribution, suggesting heavy-tailed distributions better model extreme outliers. ## Key Takeaways - **Normalized correlation** removes growth effects, revealing true time correlations - **Central Limit Theorem** generalizes random walks to any steps with defined mean/variance - **Heavy-tailed distributions** violate CLT assumptions, producing extreme outliers - **Noise as CLT**: Many small effects accumulate to normal distribution ## Mathematical Connections This lecture bridges: - [[MATH310S26-Day23-Notes|Day 23-24]]: Random walk fundamentals - [[MATH310S26-Day25-Notes|Day 25]]: Covariance and correlation - [[MATH310S26-Day22-WorkdayMaterials (Cross-correlation and signal detection)|Day 22]]: Cross-correlation in signals - Future: Recurrence relations bringing determinism back The transition from Fourier analysis → Random walks → CLT shows how deterministic and stochastic models connect through limiting processes. ## Geometric Interpretation and Unification of Correlation A useful way to understand correlation across contexts is through a geometric lens. ### Correlation as a Cosine At its core, correlation can be written as: $ \rho(X,Y) = \frac{\langle X, Y \rangle}{\|X\| \|Y\|} $ where: - $\langle X,Y \rangle$ is an [inner product](https://en.wikipedia.org/wiki/Inner_product_space) - $\|X\|$ is the induced [norm](https://en.wikipedia.org/wiki/Norm_(mathematics)) **Interpretation:** Correlation measures the [cosine](https://en.wikipedia.org/wiki/Dot_product#Geometric_definition) of the angle between two objects after removing scale. It depends only on direction, not magnitude. --- ### Sample Statistics as Geometry in $\mathbb{R}^n$ Given data: $ (x_1,\dots,x_n), \quad (y_1,\dots,y_n) $ we first center: $ \tilde{x}_i = x_i - \bar{x}, \quad \tilde{y}_i = y_i - \bar{y} $ Then correlation becomes: $ \rho = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})} {\sqrt{\sum_{i=1}^n (x_i - \bar{x})^2} \sqrt{\sum_{i=1}^n (y_i - \bar{y})^2}} $ This is exactly: $ \rho = \frac{\langle \tilde{\mathbf{x}}, \tilde{\mathbf{y}} \rangle}{\|\tilde{\mathbf{x}}\|\|\tilde{\mathbf{y}}\|} $ **Why it looks complicated:** - Data is not mean-zero → must center - No structural assumptions → all terms remain --- ### Random Walk as a Structured Geometric Object For the random walk: $ S_n = \sum_{i=1}^n X_i $ we compare $S_n$ and $S_{n+k}$. Embed both in the same space $\mathbb{R}^{n+k}$: $ S_n \leftrightarrow (1,1,\dots,1,0,\dots,0) $ $ S_{n+k} \leftrightarrow (1,1,\dots,1,1,\dots,1) $ This shows the **padding**: the earlier walk has zeros in positions corresponding to future increments. --- ### Correlation via Geometry - Inner product (overlap): $ \langle S_{n+k}, S_n \rangle = n $ - Norms: $ \|S_n\| = \sqrt{n}, \quad \|S_{n+k}\| = \sqrt{n+k} $ Thus: $ \rho_{n+k,n} = \frac{n}{\sqrt{n}\sqrt{n+k}} = \frac{\sqrt{n}}{\sqrt{n+k}} $ --- ### Interpretation: Overlap vs Total Magnitude - Numerator = shared history (first $n$ steps) - Denominator = total size of each walk $ \rho = \left(\frac{\text{shared}}{\text{total}}\right)^{1/2} $ --- ### Connection to Covariance Formulation The probabilistic definition: $ \rho = \frac{\operatorname{Cov}(S_{n+k}, S_n)}{\sigma_{S_{n+k}}\sigma_{S_n}} $ is the same computation, where: - inner product = expectation - independence removes cross terms - variance provides the norms --- ### Least Squares and $R^2$ (Projection onto a Subspace) In [linear regression](https://en.wikipedia.org/wiki/Linear_regression) (from [[MATH310S26-Day3-WorkdayMaterials (Two-by-two matrices, normal equations, and ordinary least squares)|Day 3]]), we work with a centered response vector: $ \tilde{\mathbf{y}} = \mathbf{y} - \bar{y}\mathbf{1} $ [Least squares](https://en.wikipedia.org/wiki/Least_squares) finds $\tilde{\hat{\mathbf{y}}}$ by projecting $\tilde{\mathbf{y}}$ onto the column space of $X$: $ \tilde{\mathbf{y}} = \tilde{\hat{\mathbf{y}}} + \mathbf{e}, \quad \tilde{\hat{\mathbf{y}}} \perp \mathbf{e} $ This gives the [Pythagorean identity](https://en.wikipedia.org/wiki/Pythagorean_theorem): $ \|\tilde{\mathbf{y}}\|^2 = \|\tilde{\hat{\mathbf{y}}}\|^2 + \|\mathbf{e}\|^2 $ Dividing through: $ R^2 = \frac{\|\tilde{\hat{\mathbf{y}}}\|^2}{\|\tilde{\mathbf{y}}\|^2} $ **Interpretation:** - $R^2$ ([coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination)) measures how much of $\tilde{\mathbf{y}}$ lies in the predictor subspace - Geometrically: $ R^2 = \cos^2(\theta) $ where $\theta$ is the angle between $\tilde{\mathbf{y}}$ and its projection $\tilde{\hat{\mathbf{y}}}$ **Special case (simple regression):** $ R^2 = \rho^2 $ --- ### Unifying Perspective: Hilbert Space Geometry All of these constructions live in the same framework: - We have a space of objects (vectors, data, random variables) - We define an inner product - Norms and angles follow from this structure This is the structure of a **[Hilbert space](https://en.wikipedia.org/wiki/Hilbert_space)**. In this setting: - Covariance = inner product - Correlation = cosine of angle - Regression = [orthogonal projection](https://en.wikipedia.org/wiki/Projection_(linear_algebra)) - $R^2$ = squared cosine (fraction of energy captured) --- ### Summary Table | Context | Inner product | Objects | Output | Interpretation | Course Connection | |--------|--------------|--------|--------|----------------|-------------------| | Geometry | dot product | vectors | $\cos\theta$ | alignment | Foundation | | Sample statistics | sum of products | centered data | $\rho$ | empirical alignment | [[MATH310S26-Day6-Work\|Day 6 PCA]] | | Probability | expectation | random variables | $\text{Cor}(X,Y)$ | dependence | [[MATH310S26-Day25-Notes\|Day 25]] | | Random walk | expectation + independence | sums of increments | $\frac{\sqrt{n}}{\sqrt{n+k}}$ | shared vs total history | [[MATH310S26-Day23-Notes\|Days 23]]-[[MATH310S26-Day25-Notes\|25]] | | Least squares | projection via inner product | response vs predictor subspace | $R^2$ | fraction of variance explained | [[MATH310S26-Day3-WorkdayMaterials (Two-by-two matrices, normal equations, and ordinary least squares)\|Day 3 OLS]] | --- ### Key Takeaway The difference between formulas is structural, not conceptual: - Sample statistics: no assumptions → fully expanded formula - Random walk: symmetry + independence → simplified form (see [[MATH310S26-Day24-Notes\|Day 24]]) - Regression: projection onto a subspace (see [[MATH310S26-Day3-WorkdayMaterials (Two-by-two matrices, normal equations, and ordinary least squares)\|Day 3]]) - Geometry: reveals all as the same operation **Correlation and $R^2$ are both expressions of inner product geometry:** - Correlation compares two directions - $R^2$ measures how much of one direction lies in a subspace Everything reduces to: $ \text{inner product} \;\longrightarrow\; \text{norm} \;\longrightarrow\; \text{angle} $ This unification connects: - [[MATH310S26-Day3-WorkdayMaterials (Two-by-two matrices, normal equations, and ordinary least squares)|Day 3]]: Where we first saw least squares geometry - [[MATH310S26-Day5-Notes|Day 5]]-[[MATH310S26-Day6-Work|6]]: Eigenanalysis and orthogonal decomposition in PCA - [[MATH310S26-Day7-Notes|Day 7]]-[[MATH310S26-Day10-Notes|10]]: Fourier series as orthogonal projections onto sinusoids - [[MATH310S26-Day22-WorkdayMaterials (Cross-correlation and signal detection)|Day 22]]: Cross-correlation as inner products of signals - [[MATH310S26-Day23-Notes|Day 23]]-[[MATH310S26-Day25-Notes|25]]: Random walk correlations through this geometric lens