Jacobian and Hessian Matrices

The Jacobian matrix of a vector-valued function compiles all the first-order partial derivatives. For f : \mathbb{R}^m \to \mathbb{R}^n, it helps track how each output changes in response to each input variable.

J(\mathbf{x}) = \begin{bmatrix}\frac{\partial f_1}{\partial x_1} & \frac{\partial f_1}{\partial x_2} & \cdots & \frac{\partial f_1}{\partial x_n} \\\frac{\partial f_2}{\partial x_1} & \frac{\partial f_2}{\partial x_2} & \cdots & \frac{\partial f_2}{\partial x_n} \\\vdots & \vdots & \ddots & \vdots \\\frac{\partial f_m}{\partial x_1} & \frac{\partial f_m}{\partial x_2} & \cdots & \frac{\partial f_m}{\partial x_n}\end{bmatrix}

• J(x) is the Jacobian matrix of f at point x.
• It has m rows and n columns.

Example: How to Calculate the Jacobian Matrix

Find the Jacobian matrix at the point (1,2) of the following function: f(x, y) = \begin{pmatrix}x^4 + 3 y^2 x \\5 y^2 - 2 x y + 1\end{pmatrix}

First of all, we calculate all the first-order partial derivatives of the function:

\frac{\partial f_1}{\partial x} = 4x^3 + 3y^2

\frac{\partial f_1}{\partial y} = 6yx

\frac{\partial f_2}{\partial x} = -2y

\frac{\partial f_2}{\partial y} = 10y - 2x

Now we apply the formula of the Jacobian matrix. In this case, the function has two variables and two vector components, so the Jacobian matrix will be a 2×2 square matrix:

J_f(x, y) = \begin{pmatrix}\frac{\partial f_1}{\partial x} & \frac{\partial f_1}{\partial y} \\\frac{\partial f_2}{\partial x} & \frac{\partial f_2}{\partial y}\end{pmatrix}=\begin{pmatrix}4x^3 + 3y^2 & 6yx \\-2y & 10y - 2x\end{pmatrix}

Once we have found the expression of the Jacobian matrix, we evaluate it at point (1,2):

J_f(1,2) = \begin{pmatrix}4 \cdot 1^3 + 3 \cdot 2^2 & 6 \cdot 2 \cdot 1 \\-2 \cdot 2 & 10 \cdot 2 - 2 \cdot 1\end{pmatrix}

And finally, we perform the operations:

J_f(1,2) =\begin{pmatrix}16 & 12 \\-4 & 18\end{pmatrix}

The Hessian Matrix

The Hessian matrix is an n \times n square matrix composed of the second-order partial derivatives of a function of n variables.

The formula for the Hessian matrix is:

H(f)(\mathbf{x}) = \begin{bmatrix}\frac{\partial^2 f}{\partial x_1^2} & \frac{\partial^2 f}{\partial x_1 \partial x_2} & \cdots & \frac{\partial^2 f}{\partial x_1 \partial x_n} \\\frac{\partial^2 f}{\partial x_2 \partial x_1} & \frac{\partial^2 f}{\partial x_2^2} & \cdots & \frac{\partial^2 f}{\partial x_2 \partial x_n} \\\vdots & \vdots & \ddots & \vdots \\\frac{\partial^2 f}{\partial x_n \partial x_1} & \frac{\partial^2 f}{\partial x_n \partial x_2} & \cdots & \frac{\partial^2 f}{\partial x_n^2}\end{bmatrix}

Therefore, the Hessian matrix will always be a square matrix whose dimension will be equal to the number of variables of the function. For example, if the function has 3 variables, the Hessian matrix will be a 3×3 dimension matrix.

Example: How to calculate the Hessian Matrix:

Calculate the Hessian matrix at the point (1,0) of the following multivariable function: f(x, y) = y^4 + x^3 + 3x^2 + 4y^2 - 4xy - 5y + 8

First of all we have to compute the first-order partial derivatives of the function:

\frac{\partial f}{\partial x} = 3x^2 + 6x - 4y
\frac{\partial f}{\partial y} = 4y^3 + 8y - 4x - 5

Once we know the first derivatives, we calculate all the second-order partial derivatives of the function:

\frac{\partial^2 f}{\partial x^2} = 6x + 6

\frac{\partial^2 f}{\partial y^2} = 12y^2 + 8

\frac{\partial^2 f}{\partial x \partial y} = \frac{\partial^2 f}{\partial y \partial x} = -4

Now we can find the Hessian matrix using the formula for 2×2 matrices:

H_f(x, y) = \begin{pmatrix}\frac{\partial^2 f}{\partial x^2} & \frac{\partial^2 f}{\partial x \partial y} \\\frac{\partial^2 f}{\partial y \partial x} & \frac{\partial^2 f}{\partial y^2}\end{pmatrix}=\begin{pmatrix}6x + 6 & -4 \\-4 & 12y^2 + 8\end{pmatrix}

So the Hessian matrix evaluated at the point (1,0) is:

H_f(1, 0) = \begin{pmatrix}6 \cdot 1 + 6 & -4 \\-4 & 12 \cdot 0^2 + 8\end{pmatrix}

H_f(1, 0) = \begin{pmatrix}12 & -4 \\-4 & 8\end{pmatrix}

Applications

Applications of Jacobian Matrix

• Multivariate Function Sensitivity: It captures how small changes in inputs affect multiple outputs simultaneously. It is essential for understanding sensitivity and stability in systems.
• Change of Variables in Integrals: The Jacobian determinant is critical for transforming variables in multivariable integrals, including coordinate transformations like polar, cylindrical or spherical coordinates.
• Neural Network Backpropagation: It encodes gradients of vector-valued functions, important for computing derivatives of neural network layers efficiently through automatic differentiation.
• Feature Sensitivity & Explainability: In machine learning, Jacobians provide insight into how sensitive model outputs are to each input feature.

Applications of Hessian Matrix

• Optimization Algorithms: The Hessian provides second-order derivative information that speeds up convergence in optimization. Newton’s method and Quasi-Newton methods (like L-BFGS) use Hessians or approximations.
• Curvature Analysis: Hessians quantify how the gradient changes around a point, guiding step size and direction in gradient-based optimization.
• Natural Gradient Methods: Used in advanced optimization to consider the geometry of parameter space, improving training efficiency.
• Training Large-scale Neural Networks: Techniques leverage diagonal or low-rank Hessian approximations for scalable second-order optimization.

Practice Problems

Jacobian Matrix

1. Compute the Jacobian matrix at the point (0, -2) of the following vector-valued function with 2 variables: f(x,y) = (e^{xy} + y , y^2 x)
2. Calculate the Jacobian matrix of the following 2-variable function at the point (2, -1): f(x,y)=(x^3 y^2 - 5x^2 y^2 , y^6 - 3y^3 x + 7)
3. Calculate the Jacobian matrix of the following 2-variable function at the point (2, -1): f(x, y, z) = \left( x e^{2y} \cos(-z) ,\; (y - 2)^3 \cdot \sin\left( \frac{z}{2} \right),\; e^{2y} \cdot \ln\left( \frac{x}{3} \right) \right)

Hessian Matrix

1. Find the Hessian matrix of the following 2 variable function at point (1,1): f( x , y ) = x^2 y + y^2 x
2. Calculate the Hessian matrix at the point (1,1) of the following function with two variables: f(x,y) = e^{y \ln x}
3. Compute the Hessian matrix at the point (0,1,π) of the following 3 variable function: f(x, y, z) = e^{-x} \cdot \sin(yz)