Bar 02 · The Gaussian

The multivariate normal and density matrices

When several normal variables are considered together, their joint behaviour is captured by a vector of means and a covariance matrix. This page follows that generalisation, examines the quadratic form inside the exponent, and reads the conditional and marginal laws from the matrix itself.

The joint density

A vector of variables is multivariate normal when every linear combination of its entries is normal. The law is then fixed by a mean vector and a covariance matrix, which must be symmetric and positive definite. The joint density places the quadratic form formed from the centred vector and the inverse covariance matrix inside an exponential, and carries a normalising constant built from the matrix itself.

The contours of the density are ellipses. Their shape and tilt are read from the eigenvectors and eigenvalues of the covariance matrix: the eigenvectors give the directions of the axes, and the square roots of the eigenvalues give the half-lengths. When the off-diagonal entries vanish, the ellipses stand upright and the variables are uncorrelated.

Marginal densities

Any subset of a normal vector is itself normal. Its mean vector is the corresponding part of the original mean, and its covariance matrix is the corresponding block of the original matrix read off by deleting the remaining rows and columns. This closure under taking parts is one of the most convenient features of the family, and it explains why the normal law travels so easily between dimensions.

Conditional densities

Conditioning on some entries of the vector leaves the rest normal, but with a shifted mean and a reduced matrix. The shift is linear in the held values, and the reduction no longer depends on them. The coefficient of that linear shift is built from the covariance matrix by a rule closely related to regression, and its magnitude measures how strongly the held entries speak about the remaining ones.

Bivariate density matrices

In the simplest multivariate case, two variables are gathered into a small matrix of density whose diagonal entries are the two variances and whose single off-diagonal entry is the covariance. Standardising each variable turns this into a correlation matrix with ones on the diagonal. The conditional mean of one variable given the other is then a straight line, and the conditional variance is the original variance reduced by the square of the correlation.

The relationship between the covariance matrix and the correlation matrix is taken up more fully on the next page, which treats symmetry and positive semi-definiteness as the two algebraic requirements that keep an arrangement of coefficients admissible.

Higher dimensions

The same reading carries into any number of variables. A mean becomes a longer vector, a covariance becomes a larger square arrangement, and the contours become ellipsoids whose axes follow the eigenvectors of that arrangement. Nothing in the argument changes with the count of variables; only the bookkeeping grows, and with it the care needed when the arrangement is read back into the language of conditional shifts.