## Introduction You encounter several mysterious mathematical surfaces—smooth, curved landscapes that rise and fall across two-dimensional domains. Some have peaks and valleys, others have saddle-shaped regions that curve up in one direction while curving down in another. Each surface seems to have its own unique character and behavior around critical points. These aren't just abstract mathematical objects. The techniques you'll develop for understanding these surfaces form the foundation for optimization algorithms that power modern machine learning, economic modeling, and engineering design. When a computer "learns" to recognize images or an economist models consumer behavior, the mathematics happening behind the scenes is fundamentally about navigating these kinds of curved landscapes. This project builds on our discussion of Taylor series of a single variable and matrix-vector products to you to introduce the multivariate Taylor series, which is a tool that reduces complicated functions of several variables to multivariate polynomial, just as it did in the single variable case. a powerful tool that lets you approximate complex surfaces with simple quadratic functions. ## Project Description Working with several surfaces, you will use multivariate Taylor series to create local quadratic approximations around critical points, i.e., points where the gradient of the multivariate function vanishes. Through hand calculations and geometric analysis, you'll discover how the shape of these approximations predicts the behavior of optimization algorithms. **1 Point - Foundation**: Implement multivariate Taylor series expansions through second order. Calculate gradients and Hessian matrices by hand, construct quadratic approximations, and visualize how well they match the original surfaces near critical points. Analyze the geometric meaning of Hessian eigenvalues and eigenvectors by connecting them to the shape of quadratic surfaces. Investigate how different types of critical points (minima, maxima, saddle points) manifest in the approximating quadratics and their eigenstructure. **2 Points - Exploration**: Visualization of the associated symbolic and numeric calculations. **Key Deliverable**: A geometric analysis, involving both hand calculations and computation, that demonstrates how local quadratic approximations can describe the local shape of a surface, with insights into how this mathematics enables modern optimization algorithms. ## Mathematical Background: From Curves to Surfaces The single-variable Taylor series extends naturally to multiple variables, but the geometric interpretation becomes much richer in higher dimensions. ### Single Variable Review For $f: \mathbb{R} \to \mathbb{R}$, the Taylor series around $x_0$ is: $f(x) = f(x_0) + f'(x_0)(x-x_0) + \frac{f''(x_0)}{2!}(x-x_0)^2 + \cdots$ The second-order approximation $p_2(x) = f(x_0) + f'(x_0)(x-x_0) + \frac{f''(x_0)}{2}(x-x_0)^2$ captures the local concavity. ### Multivariate Extension For $f: \mathbb{R}^2 \to \mathbb{R}$, the second-order Taylor expansion around $\mathbf{x}_0 = (x_0, y_0)$ is: $f(x,y) \approx f(\mathbf{x}_0) + \nabla f|_{\mathbf{x}_0} \cdot (\mathbf{x} - \mathbf{x}_0) + \frac{1}{2}(\mathbf{x} - \mathbf{x}_0)^T H(\mathbf{x}_0) (\mathbf{x} - \mathbf{x}_0)$ where the we have the **gradient vector** evaluated at the center point of the Taylor series $\nabla f|_{\mathbf{x}_0} = \begin{bmatrix} f_x(x_0, y_0) \\ f_y(x_0, y_0) \end{bmatrix}$ which is can also be thought of at the collection of first partial derivatives of the multivariate function. The collection of second-partial derivatives forms the **hessian matrix** $H(\mathbf{x}_0) = \begin{bmatrix} f_{xx}(x_0, y_0) & f_{xy}(x_0, y_0) \\ f_{yx}(x_0, y_0) & f_{yy}(x_0, y_0) \end{bmatrix},$ which is again evaluated at the center point of the Taylor series. Past this, we do not have a great notation for the collection of higher-order derivatives and so we must concede a local quadric surface approximation. ### Worked Example: Quadratic Surface Analysis Consider the function $f(x,y) = x^2 - xy + 2y^2$ expanded around the point $(0,0)$. **Step 1: Calculate partial derivatives** and note that $(0,0)$ is point where both first derivatives vanish, $f_x = 2x - y, \quad f_y = -x + 4y.$ We adopt the same language as single-variable calculus and call $(0,0)$ a critical point of the surface. We would like to know what the surface is doing in the neighborhood of the critical point and to find this out, we need the collection of second derivatives, $f_{xx} = 2, \quad f_{xy} = f_{yx} = -1, \quad f_{yy} = 4,$ where Clairaut's theorem tells us that the cross-partial derivatives are equivalent, since the cross derivatives are continuous. **Step 2: Evaluate at expansion point** and use it to define the quadratic multivariate Taylor series approximation of the function about $(0,0)$, $f(x,y) \approx 0 + 0 + \frac{1}{2}\begin{bmatrix} x \\ y \end{bmatrix}^T \begin{bmatrix} 2 & -1 \\ -1 & 4 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} = x^2 - xy + 2y^2$ where, $f(0,0) = 0$, $\nabla f = \begin{bmatrix} 0 \\ 0 \end{bmatrix}$, and $H = \begin{bmatrix} 2 & -1 \\ -1 & 4 \end{bmatrix}$. which is exactly what we started with, suggesting the Taylor series formulae are correct. ### Geometric Interpretation Through Eigenanalysis Since the multivariate quadratic approximation has a mixed second order term, i.e., $-xy$, it's not clear what shape is being made. However, if we consider the eigenvalues and eigenvectors of the Hessian, then we find $\det(H - \lambda I) = \det\begin{bmatrix} 2-\lambda & -1 \\ -1 & 4-\lambda \end{bmatrix} = (2-\lambda)(4-\lambda) - 1 = \lambda^2 - 6\lambda + 7$ $\lambda_1 = 3 + \sqrt{2} \approx 4.41, \quad \lambda_2 = 3 - \sqrt{2} \approx 1.59$ which tells us that the shape is concave up in the direction of its two eigenvectors, implying that $(0,0)$ is a local minimum. In fact, if we found them, we can also find that the eigenvectors are the principal axes of the elliptical level curves. - https://www.desmos.com/3d/c8tqpiwai3 - https://www.desmos.com/3d/ddmykptyof#:~:text=later%2C%20save%20and-,share,-instead. ## Implementation Guidelines ### Part 1: Manual Taylor Series Construction #### Step 1: Surface Visualization Let's start by plotting the surface for which we did the hand calculations above. **MATLAB Implementation:** ```matlab % Code for visualizing the f(x,y) from our hand calculation. f1 = @(x,y) x.^2-x.*y+2*y.^2; % Create a surface plot to visualize [X, Y] = meshgrid(-5:0.2:5, -5:0.2:5); Z1 = f1(X, Y); figure; surf(X, Y, Z1); xlabel('x'); ylabel('y'); zlabel('f(x,y)'); title('Mystery Surface 1'); shading interp; ``` **Python Implementation:** ```python import numpy as np import matplotlib.pyplot as plt from mpl_toolkits.mplot3d import Axes3D # Mystery Function 1: Don't look at the formula yet! def f1(x, y): return x**2 -x*y+2*y**2 # Create a surface plot to visualize x = np.linspace(-5, 5, 50) y = np.linspace(-5, 5, 50) X, Y = np.meshgrid(x, y) Z1 = f1(X, Y) fig = plt.figure(figsize=(10, 8)) ax = fig.add_subplot(111, projection='3d') surf = ax.plot_surface(X, Y, Z1, cmap='viridis', alpha=0.8) ax.set_xlabel('x') ax.set_ylabel('y') ax.set_zlabel('f(x,y)') ax.set_title('Mystery Surface 1') plt.colorbar(surf) plt.show() ``` #### Step 2: Hessian Analysis **MATLAB:** ```matlab % After finding critical point (x0, y0) by hand x0 = % your calculated x-coordinate y0 = % your calculated y-coordinate % Symbolic check of second derivatives by hand, then verify numerically syms x y f1_sym = x.^2-x.*y+2*y.^2; % Second partial derivatives f_xx = diff(diff(f1_sym, x), x); f_xy = diff(diff(f1_sym, x), y); f_yy = diff(diff(f1_sym, y), y); % Evaluate Hessian at critical point H = double([subs(f_xx, [x y], [x0 y0]), subs(f_xy, [x y], [x0 y0]); subs(f_xy, [x y], [x0 y0]), subs(f_yy, [x y], [x0 y0])]); fprintf('Hessian matrix at (%.2f, %.2f):\n', x0, y0); disp(H); % Calculate eigenvalues and eigenvectors by hand, then verify [eigenvectors, eigenvalues] = eig(H); disp('Eigenvalues:'); disp(diag(eigenvalues)); disp('Eigenvectors:'); disp(eigenvectors); ``` **Python:** ```python import sympy as sp # After finding critical point (x0, y0) by hand x0 = # your calculated x-coordinate y0 = # your calculated y-coordinate # Calculate second derivatives by hand, then verify numerically x, y = sp.symbols('x y') f1_sym = x**2 -x*y+2*y**2 # Second partial derivatives f_xx = sp.diff(f1_sym, x, 2) f_xy = sp.diff(f1_sym, x, y) f_yy = sp.diff(f1_sym, y, 2) # Evaluate Hessian at critical point H = np.array([[float(f_xx.subs([(x, x0), (y, y0)])), float(f_xy.subs([(x, x0), (y, y0)]))], [float(f_xy.subs([(x, x0), (y, y0)])), float(f_yy.subs([(x, x0), (y, y0)]))]]) print(f'Hessian matrix at ({x0:.2f}, {y0:.2f}):') print(H) # Calculate eigenvalues and eigenvectors by hand, then verify eigenvalues, eigenvectors = np.linalg.eig(H) print('Eigenvalues:', eigenvalues) print('Eigenvectors:') print(eigenvectors) ``` #### Step 3: Construct and Visualize Taylor Approximation Build the second-order Taylor polynomial and compare with the original: **MATLAB:** ```matlab % Construct Taylor approximation around critical point f0 = f1(x0, y0); % Function value at critical point grad = [0; 0]; % Gradient is zero at critical point % Taylor approximation function taylor_approx = @(x,y) f0 + grad(1)*(x-x0) + grad(2)*(y-y0) + ... 0.5*((x-x0).^2*H(1,1) + 2*(x-x0).*(y-y0)*H(1,2) + (y-y0).^2*H(2,2)); % Plot comparison figure; subplot(1,2,1); contour(X, Y, Z1, 20); hold on; plot(x0, y0, 'ro', 'MarkerSize', 10, 'MarkerFaceColor', 'r'); title('Original Function'); subplot(1,2,2); Z_taylor = taylor_approx(X, Y); contour(X, Y, Z_taylor, 20); hold on; plot(x0, y0, 'ro', 'MarkerSize', 10, 'MarkerFaceColor', 'r'); title('Taylor Approximation'); ``` **Python:** ```python # Construct Taylor approximation around critical point f0 = f1(x0, y0) # Function value at critical point grad = np.array([0, 0]) # Gradient is zero at critical point # Taylor approximation function def taylor_approx(x, y): dx, dy = x - x0, y - y0 return f0 + grad[0]*dx + grad[1]*dy + 0.5*(dx**2*H[0,0] + 2*dx*dy*H[0,1] + dy**2*H[1,1]) # Plot comparison fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 6)) # Original function cs1 = ax1.contour(X, Y, Z1, levels=20) ax1.plot(x0, y0, 'ro', markersize=10) ax1.set_title('Original Function') ax1.set_xlabel('x') ax1.set_ylabel('y') # Taylor approximation Z_taylor = taylor_approx(X, Y) cs2 = ax2.contour(X, Y, Z_taylor, levels=20) ax2.plot(x0, y0, 'ro', markersize=10) ax2.set_title('Taylor Approximation') ax2.set_xlabel('x') ax2.set_ylabel('y') plt.tight_layout() plt.show() ``` ## Part 4: Visualization of Eigenvectors We now consider what the eigenvectors look like relative to the elliptical contour lines of the surface. **MATLAB:** ```matlab % Visualize eigenvector directions on contour plot figure; contour(X, Y, Z_taylor, 20); hold on; plot(x0, y0, 'ro', 'MarkerSize', 10, 'MarkerFaceColor', 'r'); % Plot eigenvector directions scale = 2; % Scale factor for visibility quiver(x0, y0, scale*eigenvectors(1,1), scale*eigenvectors(2,1), ... 'r', 'LineWidth', 2, 'MaxHeadSize', 0.5); quiver(x0, y0, scale*eigenvectors(1,2), scale*eigenvectors(2,2), ... 'b', 'LineWidth', 2, 'MaxHeadSize', 0.5); legend('Contours', 'Critical Point', 'Principal Direction 1', 'Principal Direction 2'); title('Principal Curvature Directions'); ``` **Python:** ```python # Visualize eigenvector directions on contour plot plt.figure(figsize=(10, 8)) cs = plt.contour(X, Y, Z_taylor, levels=20) plt.plot(x0, y0, 'ro', markersize=10) # Plot eigenvector directions scale = 2 # Scale factor for visibility plt.quiver(x0, y0, scale*eigenvectors[0,0], scale*eigenvectors[1,0], color='red', width=0.005, scale=1, scale_units='xy', angles='xy') plt.quiver(x0, y0, scale*eigenvectors[0,1], scale*eigenvectors[1,1], color='blue', width=0.005, scale=1, scale_units='xy', angles='xy') plt.legend(['Contours', 'Critical Point', 'Principal Direction 1', 'Principal Direction 2']) plt.title('Principal Curvature Directions') plt.xlabel('x') plt.ylabel('y') plt.axis('equal') plt.show() ``` #### Student Tasks After you have implemented these codes and interpreted them relative to the original problem, adapt them to the surface $f(x,y)=\sin(x)+\sin(y)$ and proceed in the same fashion: 1. Calculate the critical points of the surface. 2. Calculate the Hessian at these critical points. - For ease of use choose the two points $(\pi/2, \pi/2)$ and $(-\pi/2,3\pi/2)$, as opposed to the infinite collection of critical points, generally. 3. Calculate the eigenvalues and eigenvectors of the Hessian matrix and using them, **predict** the nature of the surface near the critical point. 4. Run the above code twice, once for each of the two points. 5. Complete the checklist below and submit your work. ## Analysis Framework ### Geometric Insights - How do the **eigenvalue magnitudes** relate to the "sharpness" of concavity in each principal direction? - What is the geometric meaning when one eigenvalue is much larger than the other? - If you expedite your path to the top/bottom, then what does this analysis tell you about the direction that you should follow? ### Surface Classification - How do eigenvalue signs predict the behavior of optimization algorithms near critical points? - What does the **condition number** of the Hessian (ratio of largest to smallest eigenvalue) tell us about convergence rates? ## Real-World Context Multivariate Taylor series and Hessian analysis form the mathematical foundation for many modern applications: ### Machine Learning Applications - **[Gradient Descent Optimization](https://en.wikipedia.org/wiki/Gradient_descent)**: How algorithms navigate loss function landscapes to find optimal parameters - **[Newton's Method](https://www.google.com/search?q=newton+method+optimization+machine+learning)**: Using second-order information to accelerate convergence - **[Loss Function Analysis](https://www.google.com/search?q=loss+landscape+analysis+neural+networks)**: Understanding why some neural networks train faster than others ### Economic Modeling Applications - **[Utility Maximization](https://en.wikipedia.org/wiki/Utility_maximization_problem)**: How consumers make optimal choices given budget constraints - **[Production Optimization](https://www.google.com/search?q=production+function+optimization+economics)**: Finding optimal input combinations to maximize output - **[Market Equilibrium](https://en.wikipedia.org/wiki/Economic_equilibrium)**: Analyzing stability of economic systems using second-order conditions ### Engineering Applications - **[Structural Optimization](https://www.google.com/search?q=structural+optimization+engineering+hessian)**: Designing lightweight structures that meet safety requirements - **[Control System Design](https://en.wikipedia.org/wiki/Control_theory)**: Ensuring system stability using eigenvalue analysis - **[Signal Processing](https://www.google.com/search?q=quadratic+approximation+signal+processing)**: Approximating complex signals with simpler mathematical models ### Technical Challenges - **[Curse of Dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality)**: How Taylor approximations become less reliable in high-dimensional spaces - **[Numerical Stability](https://www.google.com/search?q=hessian+numerical+stability+optimization)**: Computing accurate second derivatives for large-scale problems - **[Local vs Global Optima](https://en.wikipedia.org/wiki/Local_optimum)**: Understanding when local analysis predicts global behavior ### Innovation Impact The geometric insights you're developing enable breakthrough technologies. Machine learning algorithms that recognize speech, translate languages, and drive autonomous vehicles all rely on navigating high-dimensional landscapes using principles you're learning. Economic models that predict market behavior and guide policy decisions use the same mathematical framework to understand complex multi-variable relationships. ## Deliverable Checklist - [ ] **Code Implementation** - [ ] Manual calculation of gradients and Hessians for mystery functions - [ ] Hand computation of eigenvalues and eigenvectors for 2×2 matrices - [ ] Construction and visualization of Taylor approximations - [ ] Geometric analysis of principal curvature directions - [ ] Visualization of surface, marking of critical point, plotting of the corresponding quadratic approximation, and plotting the corresponding eigenvectors of the Hessian. (Needed for 2 points) - [ ] **Analysis Report** - [ ] Classification of critical points based on eigenvalue analysis - [ ] Further geometric interpretation of Hessian eigenstructure - [ ] Connection between mathematical analysis and surface geometry - [ ] Discussion of implications for optimization algorithm design - [ ] **Communication** - [ ] One-page synthesis connecting multivariate calculus to geometric intuition - [ ] Five to seven minute video walkthrough of Taylor approximation construction and eigenvalue analysis - [ ] Clear explanation of how second-order analysis predicts surface behavior and optimization dynamics