A positive definite matrix is a symmetric square matrix where all eigenvalues are strictly positive, and the quadratic form xᵀAx > 0 for every nonzero vector x. This property ensures the matrix defines a strictly convex function and guarantees invertibility.
Positive definiteness arises in systems requiring stability, such as in multivariate statistics, optimization algorithms, and numerical methods. A matrix must be symmetric to qualify-this symmetry is essential because the definition relies on real eigenvalues and orthogonal eigenvectors.
For example, consider the 2×2 matrix [[2, 1], [1, 2]]. Its eigenvalues are 3 and 1-both positive. For any nonzero vector x = [x₁, x₂]ᵀ, the quadratic form xᵀAx equals 2x₁² + 2x₁x₂ + 2x₂², which simplifies to 2(x₁ + x₂/2)² + 1.5x₂²-a sum of squares that remains positive unless both x₁ and x₂ are zero. This confirms positive definiteness.
Every symmetric matrix can be diagonalized using an orthogonal matrix. For a matrix A to be positive definite, all entries in the diagonal matrix Λ (from A = QΛQᵀ) must be positive. This diagonalization reveals the matrix's behavior under coordinate transformation.
The quadratic form xᵀAx represents energy in physical systems or cost in optimization. When this form is always positive for nonzero x, the system has a unique minimum. This is why positive definite matrices appear in Newton's method and in training neural networks via gradient descent.
For a matrix to be positive definite, all leading principal minors must have positive determinants. In the 2×2 example above, det([2]) = 2 > 0 and det([[2,1],[1,2]]) = 3 > 0. This criterion, known as Sylvester's criterion, provides a computable test without eigenvalue calculation.
Positive definite matrices are a subset of Hermitian matrices (real symmetric matrices). They are closed under addition and scalar multiplication by positive numbers. The inverse of a positive definite matrix is also positive definite.
Common types include:
| Type | Description | Example |
|---|---|---|
| Diagonal | All off-diagonal entries zero; diagonal entries positive | [[3, 0], [0, 5]] |
| Identity | Unit matrix; all diagonal entries 1 | Iₙ |
| Gram matrix | Formed from inner products of linearly independent vectors | For v₁=[1,2], v₂=[3,4], G = [[5,11],[11,25]] |
These types often emerge in applications. Gram matrices, for instance, define kernel functions in support vector machines and Gaussian processes.
In statistics, the covariance matrix of a multivariate normal distribution must be positive definite to ensure a valid probability density function. If it were singular, the distribution would lie on a lower-dimensional subspace, making likelihood evaluation impossible.
In optimization, the Hessian matrix of a twice-differentiable function must be positive definite at a local minimum. This confirms the critical point is a strict local minimizer. For f(x,y) = x² + y², the Hessian is [[2,0],[0,2]], which is positive definite-confirming the origin is a global minimum.
Machine learning models like ridge regression rely on adding a small positive multiple of the identity matrix to the design matrix's Gram form. This ensures the resulting matrix is positive definite, enabling stable inversion during parameter estimation.
Many assume that a matrix with positive diagonal entries is automatically positive definite. This is false. Consider [[1, 3], [3, 1]]. Diagonals are positive, but eigenvalues are 4 and -2-making it indefinite. The off-diagonal elements can dominate and flip the sign of the quadratic form.
Another error is equating positive definiteness with positive semidefiniteness. A matrix is positive semidefinite if xᵀAx ≥ 0 for all x, allowing zero eigenvalues. But positive definite requires strict positivity-zero eigenvalues disqualify a matrix.
Also, positive definiteness depends on symmetry. A nonsymmetric matrix cannot be positive definite under standard definitions. Some software may report "positive definite" for nonsymmetric inputs based on the symmetric part, but this is a misuse of terminology.
Suppose you manage a portfolio of three assets with returns modeled as random variables. The covariance matrix Σ of returns must be positive definite to compute optimal weights via Markowitz's mean-variance model.
Given historical returns, compute Σ as the sample covariance. If Σ has a zero or negative eigenvalue, the optimization problem becomes unbounded or ill-conditioned. To fix this, apply regularization: Σ̂ = Σ + εI, where ε > 0. This shifts all eigenvalues upward, restoring positive definiteness.
Without this step, the optimizer may assign extreme weights to assets with near-zero variance, leading to unstable portfolios. In practice, ε is chosen empirically-common values range from 1e-6 to 1e-3 depending on data scale.
While debugging a reinforcement learning algorithm, I encountered a policy gradient update that failed due to a singular covariance matrix in the Fisher information metric. After adding a small regularization term, convergence improved dramatically. The matrix became positive definite, and the algorithm stabilized-proof that theoretical properties directly impact performance.
Positive definite matrices are closely tied to inner product spaces. They define valid metrics for distance and angle. Any positive definite matrix A can generate a weighted inner product ⟨x, y⟩_A = xᵀAy, which generalizes Euclidean geometry.
They also appear in Cholesky decomposition: any positive definite matrix A can be factored as A = LLᵀ, where L is lower triangular with positive diagonal entries. This decomposition enables efficient solving of linear systems Ax = b via forward and backward substitution.
When working with large-scale problems, Cholesky factorization is preferred over direct inversion due to numerical stability and speed. For a 1000×1000 matrix, Cholesky takes ~10⁶ operations versus ~10⁹ for matrix inversion.
Understanding the structure of positive definite matrices helps avoid computational pitfalls. Always verify symmetry and perform eigenvalue checks or determinant tests when in doubt.