# MATH310S26 - Introduction to Mathematical Modeling (Lecture Notes) ## Spring 2026 - Day 2 Notes ### Date: January 14, 2026 (01/14/26) --- ## Administrative Updates ### Check Your Understanding Assignments - Due Friday: Two Check Your Understanding problems will be turned in - Part 1: One problem completed in 10 minutes during class (live assessment) - Part 2: One problem completed ahead of time and submitted - Purpose: Provide live feedback on mathematical writing - Not intended to feel like a quiz - you know the questions ahead of time ### Lecture Reflection (LR) System - Due Sunday (flexible deadline) - Uses the HiTA system with integrated chatbot - Process: 1. Lecture PDFs are uploaded to the system 2. Students interact with AI about the material 3. Can ask questions, seek clarification, or explore extensions 4. System auto-grades upon completion - Alternatives available for students opposed to generative AI (speak with instructor) ### Friday Interactive Workday - Bring computational devices (laptop/tablet) - Be ready to work with [CSV](https://en.wikipedia.org/wiki/Comma-separated_values) files - Tools: [Python](https://en.wikipedia.org/wiki/Python_(programming_language)), [R](https://en.wikipedia.org/wiki/R_(programming_language)), [MATLAB](https://en.wikipedia.org/wiki/MATLAB), or any comfortable programming language - Focus: data processing and modeling --- ## Lecture Content: [Least Squares](https://en.wikipedia.org/wiki/Least_squares) and [Linear Regression](https://en.wikipedia.org/wiki/Linear_regression) ### Review: Homogeneous Linear Systems #### Recall from Day 1 For a [matrix](https://en.wikipedia.org/wiki/Matrix_(mathematics)) $A \in \mathbb{R}^{2 \times 2}$, if $\det(A) = 0$ ([determinant](https://en.wikipedia.org/wiki/Determinant)), then $A\mathbf{x} = \mathbf{0}$ is a [homogeneous system](https://en.wikipedia.org/wiki/System_of_linear_equations#Homogeneous_systems) with: - Non-unique solutions corresponding to points on a single [line](https://en.wikipedia.org/wiki/Line_(geometry)) through the origin - The two lines defined by the system are coincident (same line) ### Inhomogeneous Linear Systems #### When $A\mathbf{x} = \mathbf{b}$ (where $\mathbf{b} \neq \mathbf{0}$) If $\det(A) = 0$, we have two possible outcomes: 1. Infinitely many solutions: The lines are coincident but shifted - Same [slope](https://en.wikipedia.org/wiki/Slope), same [y-intercept](https://en.wikipedia.org/wiki/Y-intercept) - All points on the line are solutions 2. No solution: The lines are [parallel](https://en.wikipedia.org/wiki/Parallel_(geometry)) - Same slope, different y-intercepts - No shared points between the lines #### Key Insight - Matrix $A$ contains the orientation data (slopes) of the lines - Vector $\mathbf{b}$ determines the placement (y-intercepts) of the lines --- ## The Least Squares Problem ### Motivation: What if there's no exact solution? When dealing with real-world data, we often encounter [overdetermined systems](https://en.wikipedia.org/wiki/Overdetermined_system) - more equations than unknowns. These systems typically have no exact solution. ### Problem Setup #### Given Data Consider $m$ points in the plane: $(x, y) \in \{(x_1, y_1), (x_2, y_2), (x_3, y_3), \ldots, (x_m, y_m)\}$ #### Goal Find parameters $\beta_0$ and $\beta_1$ such that: $y_i = \beta_0 + \beta_1 x_i, \quad i = 1, 2, 3, \ldots, m$ This is the problem of fitting a [line](https://en.wikipedia.org/wiki/Line_fitting) to data points. ### Matrix Formulation #### Design Matrix We can write the system in matrix form: $\mathbf{y} = X\boldsymbol{\beta}$ Where: - $\mathbf{y} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_m \end{bmatrix}$ is the response [vector](https://en.wikipedia.org/wiki/Vector_(mathematics_and_physics)) (outputs/observables) - $X = \begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_m \end{bmatrix}$ is the [design matrix](https://en.wikipedia.org/wiki/Design_matrix) - $\boldsymbol{\beta} = \begin{bmatrix} \beta_0 \\ \beta_1 \end{bmatrix}$ are the [regression parameters](https://en.wikipedia.org/wiki/Regression_analysis#Regression_models) (model parameters/coefficients) #### The Problem - We have $m$ equations but only 2 unknowns ($\beta_0, \beta_1$) - When $m > 2$, the system is typically overdetermined - It's unlikely we can construct $\mathbf{y}$ exactly with linear combinations of columns of $X$ --- ## The Geometric Approach to Least Squares ### Cartoon Visualization (3D Space) Imagine: 1. The columns of $X$ span a 2D plane (a [subspace](https://en.wikipedia.org/wiki/Linear_subspace)) in higher-dimensional space (think [three-dimensional space](https://en.wikipedia.org/wiki/Three-dimensional_space) as a cartoon) 2. The vector $\mathbf{y}$ typically doesn't lie in this plane 3. We want to find the closest point in the plane to $\mathbf{y}$ ### [Orthogonal Projection](https://en.wikipedia.org/wiki/Projection_(linear_algebra)) The solution involves projecting $\mathbf{y}$ onto the [column space](https://en.wikipedia.org/wiki/Column_space) of $X$: 1. $\hat{\mathbf{y}}$: The projection of $\mathbf{y}$ onto the column space of $X$ 2. $\mathbf{z}$: The residual vector ($\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}}$) 3. Key property: $\mathbf{z}$ is perpendicular ([orthogonal](https://en.wikipedia.org/wiki/Orthogonality)) to the column space of $X$ ### Mathematical Development #### Orthogonality Condition Since $\mathbf{z} \perp$ column space of $X$: $\mathbf{z} \cdot \mathbf{x}_i = 0, \quad i = 1, 2$ ([dot product](https://en.wikipedia.org/wiki/Dot_product)) Where $\mathbf{x}_i$ is the $i$-th column of $X$. #### Substituting $\mathbf{z} = \mathbf{y} - \hat{\mathbf{y}}$ $(\mathbf{y} - \hat{\mathbf{y}}) \cdot \mathbf{x}_i = 0$ #### Since $\hat{\mathbf{y}} = X\hat{\boldsymbol{\beta}}$ $\mathbf{x}_i^T(\mathbf{y} - X\hat{\boldsymbol{\beta}}) = 0$ ([transpose](https://en.wikipedia.org/wiki/Transpose)) #### Collecting all equations $X^T(\mathbf{y} - X\hat{\boldsymbol{\beta}}) = \mathbf{0}$ --- ## The [Normal Equations](https://en.wikipedia.org/wiki/Normal_equation) ### Final Form Rearranging the orthogonality condition: $X^TX\hat{\boldsymbol{\beta}} = X^T\mathbf{y}$ These are called the normal equations. ### Key Results - $\hat{\boldsymbol{\beta}}$ is the [least squares solution](https://en.wikipedia.org/wiki/Least_squares#Linear_least_squares) - It minimizes $||\mathbf{z}||^2 = ||\mathbf{y} - X\boldsymbol{\beta}||^2$ (the [Euclidean norm](https://en.wikipedia.org/wiki/Norm_(mathematics)#Euclidean_norm) squared) - Even when $\mathbf{y} = X\boldsymbol{\beta}$ has no exact solution, the normal equations provide the "best" approximate solution ### Properties - The solution $\hat{\boldsymbol{\beta}}$ gives the parameters that minimize the sum of squared residuals - This is the foundation of [linear regression](https://en.wikipedia.org/wiki/Linear_regression) in [statistics](https://en.wikipedia.org/wiki/Statistics) - The method extends to higher dimensions and more complex models --- ## Check Your Understanding #2 ### Problem Statement Suppose you sampled data so that you captured all of the points on both lines: - $y = x + 1$ - $y = x - 1$ There is clearly no solution to the system of equations for finding $(x, y)$ that are on both lines (they are [parallel lines](https://en.wikipedia.org/wiki/Parallel_(geometry))). ### Tasks 1. Part (**): Form the corresponding linear system. If $A\mathbf{x} = \mathbf{b}$, then what are $a, b, c, d, b_1, b_2$ so that $A\mathbf{x} = \mathbf{b}$ corresponds to the two lines? 2. Part (*): What should the normal equations be? Solve this system. 3. Visualization: Solve the system of normal equations, plot it and the two lines. Explain why the solution to the normal equations makes sense. ### Expected Insight The least squares solution should give $y = x$ (the line exactly between the two parallel lines), which minimizes the total squared distance to both lines. --- ## Key Vocabulary ### Statistical Terms - [Design matrix](https://en.wikipedia.org/wiki/Design_matrix): Matrix $X$ containing the predictors/features - [Response variable](https://en.wikipedia.org/wiki/Dependent_and_independent_variables): The $y$ values we're trying to predict - [Regression parameters](https://en.wikipedia.org/wiki/Regression_analysis#Regression_models): The $\beta$ coefficients in our model - [Residuals](https://en.wikipedia.org/wiki/Errors_and_residuals): The differences between observed and predicted values ### Linear Algebra Terms - [Column space](https://en.wikipedia.org/wiki/Column_space): The span of the columns of a matrix - [Orthogonal projection](https://en.wikipedia.org/wiki/Projection_(linear_algebra)): Projecting a vector onto a subspace - [Normal equations](https://en.wikipedia.org/wiki/Normal_equation): The system $X^TX\boldsymbol{\beta} = X^T\mathbf{y}$ - [Overdetermined system](https://en.wikipedia.org/wiki/Overdetermined_system): More equations than unknowns --- ## Course Website and Resources ### Instructor Website - URL: [scottastrong.org](https://scottastrong.org) - Contains course materials and auto-generated notes - Notes are created using local LLM with lecture PDFs and transcripts - Includes Wikipedia links for additional context ### Canvas Integration - Lecture PDFs always available in Canvas files - HiTA system for lecture reflections - Check Your Understanding assignments --- ## Looking Ahead ### Friday's Interactive Workday - Data analysis with CSV files - Implementation of least squares fitting - Practice with computational tools (Python/R/MATLAB) - Discussion of A and B level projects for mathematical modeling ### Important Reminders - Check Your Understanding due Friday (one in-class, one ahead of time) - Lecture Reflection due Sunday (flexible) - Bio slides and sticker prompts should be submitted if not already done --- ## Additional Notes ### Gram–Schmidt Process - Mentioned as particularly challenging for dimensions > 3 - Success rate: ~1% once normalizing vectors - A reminder of the computational challenges in [linear algebra](https://en.wikipedia.org/wiki/Linear_algebra) - See the [Gram–Schmidt process](https://en.wikipedia.org/wiki/Gram%E2%80%93Schmidt_process) for details