# MATH310S26 - Introduction to Mathematical Modeling (Lecture Notes) ## Spring 2026 - Day 3 Notes ### Date: January 21, 2026 (01/21/26) --- ## Administrative Updates ### Bio Slides - Feedback has been given to bio slides with comments on student interests - Working on creating a summary of class interests - Final deck to be posted by tomorrow morning - Last chance to make any changes to your slide ### Additional Activity 01 - Due: Upload working codes from Day 3 exercises - Requirements: 1. Code files 2. Visualizations 3. Results (including coefficient of determination and 3D visualization) - Submit via Canvas with text entry for results ### Upcoming Schedule - Monday (Day 4): Interactive workday - Continue working with the 3D dataset - Discuss context and meaning of the data - Wrap up linear algebra operations for the course - Review 2×2 matrices, ordinary least squares, and eigenvalues --- ## Lecture Content: Least Squares Review and Mean Centering ### Review: Least Squares Problem Setup #### Given Data - $m$ points in $\mathbb{R}^2$: $(x_i, y_i)$ for $i = 1, 2, ..., m$ - Goal: Find $\beta_0, \beta_1 \in \mathbb{R}$ such that: $y_i = \beta_0 + \beta_1 x_i$ #### Matrix Formulation $X\boldsymbol{\beta} = \mathbf{y}$ Where: - $X$ is the [design matrix](https://en.wikipedia.org/wiki/Design_matrix) - $\boldsymbol{\beta} = \begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix}$ are the regression parameters - $\mathbf{y}$ is the response vector #### The [Normal Equations](https://en.wikipedia.org/wiki/Normal_equation) When the system has no exact solution (likely for [overdetermined systems](https://en.wikipedia.org/wiki/Overdetermined_system)), we solve: $X^TX\hat{\boldsymbol{\beta}} = X^T\mathbf{y}$ The solution $\hat{\boldsymbol{\beta}}$ is the [least squares solution](https://en.wikipedia.org/wiki/Least_squares#Linear_least_squares) that minimizes $||\mathbf{y} - X\boldsymbol{\beta}||^2$ --- ## Feature Extension and Overfitting ### Linear in Parameters, Not Variables The fundamental equation can be extended: $y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \beta_3 x_i^3 + ...$ Key insight: This remains **linear in the parameters** $\beta_j$ even though it's polynomial in $x$. ### The Overfitting Problem When adding more features: - Line fit requires 2 points minimum - Parabola requires 3 points minimum - Cubic requires 4 points minimum **Warning**: With $n$ data points, a polynomial of degree $n-1$ will interpolate exactly through all points. **Problem**: [Overfitting](https://en.wikipedia.org/wiki/Overfitting) - The model captures the points but "misses the message" - Creates wild oscillations between data points - Poor predictive performance on new data - Related to [Runge's phenomenon](https://en.wikipedia.org/wiki/Runge%27s_phenomenon) in interpolation --- ## Coefficient of Determination (R²) ### Definition $R^2 = 1 - \frac{RSS}{TSS}$ Where: - **RSS** (Residual Sum of Squares): $\sum_i (y_i - \hat{y}_i)^2$ - **TSS** (Total Sum of Squares): $\sum_i (y_i - \bar{y})^2$ - $y_i$: Predicted values from the LSP - Note, that this is not consistent with our previous board notations, because I didn't update the slide. 😭 The real point is the behavior of the coefficient of determination $R^{2}$ discussion below. - Here we mean, $y_i = \hat{\beta}_{0}+\hat{\beta}_{1} x_i$ - $\hat{y}_i$: Observed values from the model - Note, that this is not consistent with our previous board notations, because I didn't update the slide. 😭 The real point is the behavior of the coefficient of determination $R^{2}$ discussion below. - Here we mean the sampled data output, $(\hat{x}_{i},\hat{y}_{i})$ for $i=1,2,3,4$. - $\bar{y}$: Mean of observed values ### Interpretation **When $R^2 = 1$**: - The predictions $y_i$ must be the same as observed data $\hat{y}_{i}$. - Perfect fit - RSS = 0 $\implies$ $R^2=1-\frac{RSS}{TSS}=1$ **When $R^2 = 0$**: - Let observations $\hat{y}_{i}$ correspond to predictions $y_i$, then RSS = TSS - Model performs no better than predicting the mean - Line of best fit is horizontal - No [correlation](https://en.wikipedia.org/wiki/Correlation) between variables --- ## Mean Centering of Data ### Motivation Mean centering simplifies computations and reveals structure in the data. ### Process Given original data vectors $\mathbf{x}$ and $\mathbf{y}$, create mean-centered versions: $\tilde{\mathbf{x}} = \mathbf{x} - \bar{x}\mathbf{1}$ $\tilde{\mathbf{y}} = \mathbf{y} - \bar{y}\mathbf{1}$ Where: - $\bar{x}, \bar{y}$ are the means of the respective vectors - $\mathbf{1}$ is the vector of ones - $\tilde{\mathbf{x}}, \tilde{\mathbf{y}}$ are the mean-centered data ### Example with Toy Dataset **Original data**: $(2,1), (5,2), (7,3), (8,3)$ **Means**: $\bar{x} = 5.5$, $\bar{y} = 2.25$ **Mean-centered data**: $(-3.5, -1.25), (-0.5, -0.25), (1.5, 0.75), (2.5, 0.75)$ --- ## Effects of Mean Centering on Normal Equations ### Modified Normal Equations $\tilde{X}^T\tilde{X}\hat{\boldsymbol{\beta}} = \tilde{X}^T\tilde{\mathbf{y}}$ ### Key Result: Diagonal Structure After mean centering: $\tilde{X}^T\tilde{X} = \begin{bmatrix} 4 & 0 \\ 0 & 21 \end{bmatrix}$ $\tilde{X}^T\tilde{\mathbf{y}} = \begin{bmatrix} 0 \\ 7.5 \end{bmatrix}$ ### Implications 1. **Decoupled equations**: No $\beta_1$ in the first equation, no $\beta_0$ in the second 2. **Y-intercept vanishes**: $\beta_0 = 0$ always 3. **Slope unchanged**: Same slope as the original data 4. **$R^2$ unchanged**: Same coefficient of determination ### Mathematical Explanation The off-diagonal zeros appear because: - $\mathbf{1}^T\tilde{\mathbf{x}} = \sum_i \tilde{x}_i = 0$ (mean of centered data is zero) - $\mathbf{1}^T\tilde{\mathbf{y}} = \sum_i \tilde{y}_i = 0$ (mean of centered data is zero) This ensures that the intercept term $\beta_0 = 0$ for mean-centered data. --- ## Preview: Eigenanalysis for Dimensional Reduction ### Connection to the 3D Dataset For data in $\mathbb{R}^3$ that is "flat like a pancake": - After mean centering, data passes through the origin - [Eigenvectors](https://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors) point in fundamental directions of variation - [Eigenvalues](https://en-wikipedia.org/wiki/Eigenvalues_and_eigenvectors) quantify variance in each direction ### Key Insight If two eigenvalues are much larger than the third: - The small eigenvalue corresponds to the "thickness" direction - The large eigenvalues correspond to the principal plane - Can perform [dimensionality reduction](https://en.wikipedia.org/wiki/Dimensionality_reduction) by ignoring the small eigenvalue direction This is the foundation of [Principal Component Analysis (PCA)](https://en.wikipedia.org/wiki/Principal_component_analysis). --- ## Key Takeaways ### Mean Centering Benefits 1. Simplifies normal equations to diagonal form 2. Eliminates intercept term 3. Preserves slope and $R^2$ 4. Prepares data for eigenanalysis ### Statistical Invariance - [Translation](https://en.wikipedia.org/wiki/Translation_(geometry)) of coordinate system doesn't change: - Slope of best fit line - Coefficient of determination - Relative relationships in data ### Looking Ahead - Monday: Continue with 3D dataset analysis - Apply eigenanalysis for dimensional reduction - Understand the geometric meaning of eigenvalues in data analysis --- ## Check your understanding Given $ A=\begin{bmatrix}0 & 1\\ 1 & 0\end{bmatrix}, $ and recall that **eigenvectors** are $ x\in\mathbb{R}^2 \text{ such that } Ax=\lambda x, $ where $\lambda\in\mathbb{R}$ is called the associated **eigenvalue**. 1. Find $\lambda$ such that $ (A-\lambda I)x=0 $ has infinitely many solutions. 2. Using the eigenvalues found in part 1, find the corresponding eigenvectors of $A$, normalize them, and show that they are orthogonal (perpendicular) to each other. --- [^1]: Handwritten heading written in green ink. ### Computational Exercise With the 3D dataset: 1. Perform linear regression on first two columns 2. Visualize the data in 3D space 3. Report coefficient of determination 4. Submit code and results via Canvas (Additional Activity 01) --- ## Important Reminders - Bio slide deck posting tomorrow morning - Additional Activity 01 due (code, visualizations, results) - Monday: Interactive workday with 3D dataset - Bring computational devices (laptop/tablet) on Monday