SVD: Singular Value Decomposition

What the Singular Value Decomposition Is

A matrix mixes coordinates. Every entry of the output is assembled from every entry of the input, so almost nothing worth knowing about

A

can be read from

A

itself. What is

A^{10}

? Multiply ten times and find out. What happens to a vector pushed through

A

a thousand times — does it grow, shrink, settle into some direction? There is no answer short of running it.

Factorization removes the obstacle. Split

A

into factors that each do one simple thing, and questions that required arithmetic can be settled by inspection.

Diagonalization and Its Requirement

Diagonalization does exactly that. Writing

A = PDP^{-1}

amounts to changing coordinates so that the mixing is absent: the columns of

P

are directions the matrix only stretches,

D

holds the stretching factors, and in that basis no coordinate influences any other.

Take

B = \begin{pmatrix} 2 & 1 \\ 0 & 1 \end{pmatrix}

.

Its eigenvalues are

2

and

1

, with eigenvectors

(1, 0)

and

(1, -1)

, so

B = PDP^{-1}, \qquad P = \begin{pmatrix} 1 & 1 \\ 0 & -1 \end{pmatrix}, \qquad D = \begin{pmatrix} 2 & 0 \\ 0 & 1 \end{pmatrix}

The tenth power then costs nothing:

B^{10} = PD^{10}P^{-1}

, and

D^{10} = \begin{pmatrix} 1024 & 0 \\ 0 & 1 \end{pmatrix}

because

2^{10} = 1024

. Restoring the original coordinates gives

B^{10} = \begin{pmatrix} 1024 & 1023 \\ 0 & 1 \end{pmatrix}

without a single matrix multiplication. The inverse is

PD^{-1}P^{-1}

, with no elimination anywhere. Long-run behaviour is governed by whichever

\lambda

is largest, here

2

. Nothing is approximated and nothing is recomputed — the work was done once, when the factorization was found.

That performance comes with a condition, and the condition is visible in the formula. The same matrix

P

stands on both sides of

A = PDP^{-1}

: its columns are used to read the input and again to write the output. The factorization requires that both ends be described in identical terms — one basis, serving twice.

Two matrices, two ways the requirement fails.

A

3 \times 2

matrix takes vectors from

\mathbb{R}^2

and returns vectors in

\mathbb{R}^3

, so no single basis can describe both ends, and the question of an eigenvector cannot even be posed:

A\mathbf{v} = \lambda\mathbf{v}

compares vectors living in different spaces. A defective matrix fails otherwise. It is square, but every eigenvector is a multiple of one direction, so no basis of eigenvectors exists and

P

cannot be assembled. Rectangular shape excludes diagonalization in principle, defective structure excludes it in practice, and between them they account for most matrices anyone meets.

The second failure takes three lines to verify.

Change one entry of

B

and take

A = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}

. Its characteristic polynomial is

\det(A - \lambda I) = (1 - \lambda)^2

so

\lambda = 1

is a repeated root. Solving

(A - I)\mathbf{v} = \mathbf{0}

means solving

\begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}\mathbf{v} = \mathbf{0}

, which forces

v_2 = 0

and leaves

v_1

free. Every eigenvector is a multiple of

(1, 0)

: one direction, in two dimensions. There is no second column for

P

, so

A

has no diagonalization — not because the arithmetic is hard, but because the object being asked for does not exist.

Two Bases Instead of One

Nothing forced the two bases to coincide. That was an additional demand, and it can be withdrawn. Admit one orthonormal basis for the input space and a separate one for the output space, and require only that the matrix carry the first to the second with a scaling in between. Equal dimensions are then unnecessary, fixed directions are unnecessary, and both obstructions vanish together.

The unit circle becomes an ellipse. Each

\mathbf{v}_i

is carried to a multiple of

\mathbf{u}_i

, and

\sigma_i

is that multiple.

The picture states the entire factorization in one equation. There is an orthonormal basis

\mathbf{v}_1, \dots, \mathbf{v}_n

of the input space and an orthonormal basis

\mathbf{u}_1, \dots, \mathbf{u}_m

of the output space satisfying

A\mathbf{v}_i = \sigma_i \mathbf{u}_i

for every

i

. Each input basis vector is sent to a multiple of a single output basis vector, and to nothing else.

The matrix that had no diagonalization has this one.

For the same

A = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}

, form

A^TA = \begin{pmatrix} 1 & 1 \\ 1 & 2 \end{pmatrix}, \qquad \det(A^TA - \lambda I) = \lambda^2 - 3\lambda + 1

whose roots are

\lambda = \frac{3 \pm \sqrt{5}}{2}

. The scaling factors are their square roots,

\sigma_1 = \frac{1 + \sqrt{5}}{2} \approx 1.618, \qquad \sigma_2 = \frac{\sqrt{5} - 1}{2} \approx 0.618

Two of them, both real, both positive, for a matrix with only one eigenvector. As a check,

\sigma_1\sigma_2 = 1

, which is

|\det A|

— the area scaling has to agree however it is computed. The eigenvectors of

A^TA

supply

\mathbf{v}_1

and

\mathbf{v}_2

, and

\mathbf{u}_i = A\mathbf{v}_i/\sigma_i

supplies the other basis.

The definition

Collecting those

n

statements into one matrix equation gives the singular value decomposition: an orthonormal input basis in

V

, an orthonormal output basis in

U

, and the scaling factors between them along the diagonal of

\Sigma

.

SVD

A = U\Sigma V^T

Svd →

It is a factorization, not an approximation: the three matrices multiply back to

A

exactly, with no error term and no conditions attached.

U

is

m \times m

orthogonal and its columns are the left singular vectors.

V

is

n \times n

orthogonal and its columns are the right singular vectors.

\Sigma

is

m \times n

, zero everywhere off the diagonal, carrying

\sigma_1 \geq \sigma_2 \geq \cdots \geq 0

along it. Those are the singular values, and they are the lengths of the semi-axes of the ellipse above.

What Follows and What Does Not

Several properties follow from the construction rather than from any further assumption. Both bases are orthonormal rather than merely independent, so

U^{-1} = U^T

and

V^{-1} = V^T

and neither factor is ever inverted. The scaling factors are real and non-negative, since they measure lengths, and can therefore be ordered from largest to smallest — an ordering no set of eigenvalues admits, eigenvalues being possibly complex. And the factorization exists for every matrix without exception: any shape, any rank, symmetric or not, invertible or not.

The same relaxation imposes a limitation, and the algebra shows exactly where.

Applying the matrix twice gives

A^2 = U\Sigma V^T \cdot U\Sigma V^T

and everything depends on the interior

V^TU

. Under diagonalization the corresponding product is

P^{-1}P = I

, which cancels and leaves

PD^2P^{-1}

. Here

V

and

U

are different orthogonal matrices, so

V^TU

is not the identity but a rotation — for the shear above, by about

26.6^{\circ}

— and nothing collapses. The question this section opened with, the tenth power and the long run, is therefore the one question the singular value decomposition cannot answer and diagonalization can. The two factorizations address different questions: one invariant directions, the other stretching. What the three factors do to a vector geometrically is the subject of the next section.

The Geometric Interpretation

Every linear transformation

\mathbf{x} \mapsto A\mathbf{x}

decomposes into three geometric steps:

V^T

rotates (or reflects) the input space, aligning the input with the "natural axes" of the transformation — the directions along which

A

stretches most and least.

\Sigma

scales each axis independently by the corresponding singular value. Axes with

\sigma_i = 0

are annihilated — those directions are collapsed to zero.

U

rotates (or reflects) the scaled result into the output space.

The singular values measure the stretching in each orthogonal direction.

\sigma_1

is the maximum stretching:

\sigma_1 = \max_{\|\mathbf{x}\|=1}\|A\mathbf{x}\|

. The smallest nonzero singular value

\sigma_r

is the minimum stretching on the row space. The ratio

\sigma_1/\sigma_r

is the condition number — it measures how distorted the transformation is.

Even the most complex-looking matrix is geometrically just two rotations sandwiching a coordinate-axis scaling.

SVD Notation

Notation

SVD Notation

Letters that are contracts, a summation sign moonlighting as a matrix, and the lowercase sigma with a set-valued cousin one chapter away.

A^{T}

and matrix products — matrix notation;

\lambda

and

\sigma(A)

— eigenvalue notation;

\|\cdot\|

— norm notation.

A = U\Sigma V^{T}

A equals U, Sigma, V-transpose

In decomposition notation the letters are the theorem: writing

U\Sigma V^{T}

asserts

U

and

V

orthogonal and

\Sigma

diagonal — three contracts in three glyphs. The whole family works this way:

LU

promises triangles,

QR

promises orthogonal-times-triangular,

PDP^{-1}

promises a diagonal

D

.

CasesThe contracts extend to the derived marks: the pseudoinverse of The Pseudoinverse below inverts the letters that can be inverted —

A^{+} = V\Sigma^{+}U^{T}

— with the superscript

+

joining

n

,

-1

,

T

and

\perp

in the crowded label corner.

Also written

U\Sigma V^{*}

over the complex numbers, the star being the conjugate transpose — the modern adjoint of the adjugate's naming hazard.

Do not confuseThe family's own letters.

LU

's

U

is upper triangular; SVD's

U

is orthogonal — the same letter signs two different contracts two pages apart, and only the decomposition's name says which is in force.

\Sigma

Sigma — here, a diagonal matrix

Capital sigma as an object, not an operator: the rectangular diagonal matrix carrying the singular values, doing the scaling step of The Geometric Interpretation above. Maximum-contrast glyph reuse — everywhere else on this site,

\Sigma

sums things.

CasesShape bookkeeping: for an

m \times n

matrix

A

,

\Sigma

is

m \times n

— diagonal entries

\sigma_1, \ldots, \sigma_r

, then zeros padding to the rectangle. The compact form of Compact and Thin Forms below trims the padding.

Also written

\operatorname{diag}(\sigma_1, \ldots, \sigma_r)

— the constructor spelling, with the shape stated alongside;

D

or

S

in software documentation, where sigma is hard to type.

Do not confuseThe summation sign. No bounds above or below, no summand after — a bare

\Sigma

between two matrix letters is a factor, not an instruction to add.

\sigma_1 \geq \sigma_2 \geq \cdots \geq 0

The singular values, in decreasing order

Lowercase sigma with an index — and the ordering is part of the notation: writing

\sigma_1

means the largest, by the standing convention of Singular Values below. No other subscripted family on this site carries a built-in sort.

CasesThe extremes have second names:

\sigma_1 = \|A\|_2

, the operator norm, and

\sigma_1/\sigma_r = \kappa(A)

, the condition number — kappa being numerical analysis's own reserved letter.

Also written

s_i

in some numerical texts and most software output, freeing sigma for statistics.

Do not confuseThe set-valued cousin:

\sigma(A)

with no subscript is the spectrum — the set of eigenvalues. Subscripted sigma is a singular value, bare sigma-of is a set; eigenvalues and singular values agree only for special matrices, so the two sigmas rarely even hold the same numbers.

All linear algebra symbols →↑ Decompositions

Singular Values

The singular values of

A

are the square roots of the eigenvalues of

A^TA

(or equivalently

AA^T

):

Singular Values

\sigma_i = \sqrt{\lambda_i(A^TA)} = \sqrt{\lambda_i(AA^T)}

Singular values →

Since

A^TA

is symmetric positive semi-definite, its eigenvalues are all

\geq 0

, so the singular values are real and non-negative. They are ordered

\sigma_1 \geq \sigma_2 \geq \cdots \geq 0

.

The number of nonzero singular values equals the rank of

A

:

SVD Rank

\text{rank}(A) = \#\{i : \sigma_i > 0\}

Svd rank →

This is the most numerically stable method for determining rank: compute the SVD and count singular values above a tolerance.

The largest singular value

\sigma_1

is the operator norm

\|A\|_2 = \max_{\|\mathbf{x}\|=1}\|A\mathbf{x}\|

. The Frobenius norm is

\|A\|_F = \sqrt{\sigma_1^2 + \sigma_2^2 + \cdots + \sigma_r^2}

. The condition number is

\kappa(A) = \sigma_1/\sigma_r

— a large condition number means the matrix is nearly singular and small perturbations in the input cause large changes in the output.

Singular values as roots of eigenvalues

The eigenvalues of the symmetric product have been found and their square roots taken. They are never negative, because that product cannot have negative eigenvalues, and they are listed largest first by convention. This is why every matrix has singular values even when it has no eigenvalues of its own. Compute them for a rectangular matrix on the SVD visualizer.

Their ordering is what makes truncation meaningful: cutting the list short always discards the least important part first.

Computing the SVD

The standard approach computes the SVD through the eigenvalue decomposition of

A^TA

.

Form

A^TA

(symmetric,

n \times n

). Find its eigenvalues

\lambda_1 \geq \cdots \geq \lambda_n \geq 0

and orthonormal eigenvectors

\mathbf{v}_1, \dots, \mathbf{v}_n

using the spectral decomposition. These are the right singular vectors:

V = [\mathbf{v}_1 \; \cdots \; \mathbf{v}_n]

.

The singular values are

\sigma_i = \sqrt{\lambda_i}

. The left singular vectors are computed from the right ones:

\mathbf{u}_i = \frac{1}{\sigma_i}A\mathbf{v}_i

for each nonzero

\sigma_i

. If

r < m

, extend

\{\mathbf{u}_1, \dots, \mathbf{u}_r\}

to an orthonormal basis for

\mathbb{R}^m

.

Worked Example

For

A = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{pmatrix}

:

A^TA = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}

, eigenvalues

3

and

1

, eigenvectors

\frac{1}{\sqrt{2}}(1, 1)^T

and

\frac{1}{\sqrt{2}}(1, -1)^T

. Singular values:

\sqrt{3}

and

1

. Left singular vectors:

\mathbf{u}_1 = \frac{1}{\sqrt{3}}A\mathbf{v}_1 = \frac{1}{\sqrt{6}}(1, 1, 2)^T

,

\mathbf{u}_2 = A\mathbf{v}_2 = \frac{1}{\sqrt{2}}(1, -1, 0)^T

. Extend with

\mathbf{u}_3 = \frac{1}{\sqrt{3}}(-1, -1, 1)^T

.

The three factors assembled

Two orthogonal matrices with a rectangular diagonal between them, and the whole product reconstructs the original. Unlike diagonalisation this asks nothing of the matrix — not squareness, not symmetry, not a full set of eigenvectors. Every matrix without exception has this factorisation. Build it for a matrix of your own on the SVD visualizer.

That universality is why the SVD, rather than the eigendecomposition, is the factorisation numerical work reaches for by default.

Compact and Thin Forms

The full SVD has

U

of size

m \times m

,

\Sigma

of size

m \times n

, and

V

of size

n \times n

. Two economical alternatives retain only the essential information.

The thin SVD keeps only the first

n

columns of

U

(call them

U_1

) and the top

n \times n

block of

\Sigma

(call it

\Sigma_1

):

A = U_1 \Sigma_1 V^T

. This drops the columns of

U

corresponding to the left null space.

The compact SVD keeps only the first

r

columns of

U

and

V

(where

r = \text{rank}(A)

) and the

r \times r

diagonal block of nonzero singular values:

A = U_r \Sigma_r V_r^T

. This is the most economical representation — it captures only the rank-

r

content of

A

, discarding everything associated with zero singular values.

All three forms represent the same matrix

A

. The compact form uses the least storage; the full form provides bases for all four fundamental subspaces. The three variants line up cleanly on factor dimensions and on what each one chooses to keep.

Download

PNGRaster · 2×PDFPrint quality

Decompositions · SVD

Three forms, one factorization

All three reconstruct $A$ exactly. They differ in how much of $U$ and $\Sigma$ is kept — and what is dropped is precisely the part that multiplies against zero singular values.

3forms

Everything kept1

Full SVD§ 1

U

and

V

both square and orthogonal

shapes

U: m \times m

,

\Sigma: m \times n

,

V: n \times n

what is keptall of

U

,

V

, and a rectangular

\Sigma

what is droppednothing

used fortheory, the four subspaces

The only form where

U

and

V

are genuinely orthogonal matrices, so

UU^{\mathsf{T}} = I

as well as

U^{\mathsf{T}}U = I

. That is what makes it the form to state results with — the last

m - r

columns of

U

span the left null space and would otherwise be gone.

Trimmed to what multiplies2

Thin SVD§ 5

m > n

— keep

n

columns of

U

shapes

U_n: m \times n

,

\Sigma_n: n \times n

,

V: n \times n

what is keptthe first

n

columns of

U

what is droppedthe last

m - n

columns of

U

used forleast squares on tall matrices

The dropped columns multiply rows of

\Sigma

that are entirely zero, so removing them changes nothing about the product. Note

U_n

is no longer square:

U_n^{\mathsf{T}}U_n = I_n

still holds, but

U_nU_n^{\mathsf{T}}

does not.

Compact SVD§ 5keep only the

r

nonzero singular values

shapes

U_r: m \times r

,

\Sigma_r: r \times r

,

V_r: n \times r

what is keptthe rank-

r

content only

what is droppedevery zero singular value and its vectors

used forstorage, rank-deficient matrices

The most economical exact form:

\Sigma_r

is square with a strictly positive diagonal, so it is invertible — which is what makes the pseudoinverse computable as

V_r\Sigma_r^{-1}U_r^{\mathsf{T}}

.

Nothing is approximated anywhere in this table. The full form carries orthonormal bases for all four fundamental subspaces, which is why it is the one to state theorems with; the compact form carries only what contributes to

A

, which is why it is the one to store. Approximation begins only when the compact form is truncated below rank

r

.

Three forms, one factorization·/linear-algebra/decompositions/svdLearn Math Class

The distinction to keep hold of is that none of these is an approximation. Each drops columns that multiply against zero singular values, so the product is unchanged and

A

is reconstructed exactly in all three. What is lost is not accuracy but the orthogonality of

U

as a square matrix —

U_r^{\mathsf{T}}U_r = I_r

still holds, but

U_rU_r^{\mathsf{T}}

no longer does, and any argument relying on the left null space needs the full form.

SVD and the Four Fundamental Subspaces

The SVD simultaneously provides orthonormal bases for all four fundamental subspaces of

A

:

SVD Four Fundamental Subspaces

\begin{aligned} \text{Col}(A) &= \text{Span}\{\mathbf{u}_1, \ldots, \mathbf{u}_r\} \\ \text{Null}(A^T) &= \text{Span}\{\mathbf{u}_{r+1}, \ldots, \mathbf{u}_m\} \\ \text{Row}(A) &= \text{Span}\{\mathbf{v}_1, \ldots, \mathbf{v}_r\} \\ \text{Null}(A) &= \text{Span}\{\mathbf{v}_{r+1}, \ldots, \mathbf{v}_n\} \end{aligned}

Svd four fundamental subspaces →

The first

r

columns of

V

(

\mathbf{v}_1, \dots, \mathbf{v}_r

) form an orthonormal basis for the row space of

A

.

The last

n - r

columns of

V

(

\mathbf{v}_{r+1}, \dots, \mathbf{v}_n

) form an orthonormal basis for the null space of

A

.

The first

r

columns of

U

(

\mathbf{u}_1, \dots, \mathbf{u}_r

) form an orthonormal basis for the column space of

A

.

The last

m - r

columns of

U

(

\mathbf{u}_{r+1}, \dots, \mathbf{u}_m

) form an orthonormal basis for the left null space of

A

.

No other factorization provides all four bases simultaneously, and no other method guarantees that these bases are orthonormal. The SVD is the complete structural portrait of any matrix.

Fundamental subspace	Basis from	Column indices	Dimension
Row space of A	V	1, ..., r	r
Null space of A	V	r+1, ..., n	n − r
Column space of A	U	1, ..., r	r
Left null space of A	U	r+1, ..., m	m − r

The Pseudoinverse

The Moore-Penrose pseudoinverse

A^+

is computed directly from the SVD:

Moore-Penrose Pseudoinverse

A^+ = V\Sigma^+ U^T

Moore penrose pseudoinverse →

The matrix

\Sigma^+

is formed by reciprocating each nonzero singular value and transposing the shape: if

\Sigma

is

m \times n

with diagonal entries

\sigma_1, \dots, \sigma_r, 0, \dots, 0

, then

\Sigma^+

is

n \times m

with diagonal entries

1/\sigma_1, \dots, 1/\sigma_r, 0, \dots, 0

.

The pseudoinverse satisfies four defining properties:

AA^+A = A

,

A^+AA^+ = A^+

,

(AA^+)^T = AA^+

,

(A^+A)^T = A^+A

.

For a full-rank overdetermined system (

m > n

, rank

= n

),

A^+\mathbf{b}

gives the least-squares solution. For a rank-deficient system,

A^+\mathbf{b}

gives the minimum-norm least-squares solution — the solution of smallest length among all minimizers of

\|A\mathbf{x} - \mathbf{b}\|

.

Low-Rank Approximation

The best rank-

k

approximation to

A

in either the operator norm or the Frobenius norm is obtained by truncating the SVD at

k

terms:

Eckart-Young Low-Rank Approximation

A_k = \sum_{i=1}^{k} \sigma_i \mathbf{u}_i \mathbf{v}_i^T, \qquad \|A - A_k\|_2 = \sigma_{k+1}, \quad \|A - A_k\|_F = \sqrt{\sum_{i=k+1}^{r}\sigma_i^2}

Eckart young low rank approximation →

This is the Eckart-Young-Mirsky theorem. Among all matrices of rank at most

k

,

A_k

is the closest to

A

. The approximation error equals

\sigma_{k+1}

(the first discarded singular value) in the operator norm, and

\sqrt{\sigma_{k+1}^2 + \cdots + \sigma_r^2}

in the Frobenius norm.

When the singular values decay rapidly —

\sigma_1 \gg \sigma_2 \gg \cdots

— a small number of terms captures most of the matrix. This is the basis of image compression (store

k

singular value triples instead of

mn

entries), noise reduction (discard small singular values as noise), latent semantic analysis (retain the top-

k

"concepts" in a document-term matrix), and dimensionality reduction more broadly.

SVD and Norms

The singular values provide the complete "size profile" of a matrix.

The operator (spectral) norm is the largest singular value:

Operator Norm

\|A\|_2 = \sigma_1 = \max_{\|\mathbf{x}\|=1} \|A\mathbf{x}\|

Operator norm →

It measures the maximum factor by which

A

can stretch a unit vector.

The Frobenius norm is the root-sum-of-squares of all singular values:

Frobenius Norm via Singular Values

\|A\|_F = \sqrt{\sum_{i=1}^{r} \sigma_i^2}

Frobenius norm via singular values →

It measures the total "energy" in the matrix.

The condition number quantifies sensitivity to perturbation:

Condition Number

\kappa(A) = \frac{\sigma_1}{\sigma_r}

Condition number →

A matrix with

\kappa = 10^k

loses roughly

k

digits of accuracy in solving

A\mathbf{x} = \mathbf{b}

with floating-point arithmetic. A perfectly conditioned matrix (

\kappa = 1

) is orthogonal. A singular matrix (

\sigma_r = 0

) has

\kappa = \infty

.

The singular values are the natural measuring tool for matrices, just as eigenvalues are the natural measuring tool for symmetric matrices and linear operators. For non-symmetric matrices, singular values (not eigenvalues) govern norms and conditioning.

Quantity	Formula via singular values	Interpretation
Operator (spectral) norm ‖A‖₂	σ₁	maximum stretching factor on the unit ball
Frobenius norm ‖A‖_F	√(σ₁² + σ₂² + ··· + σ_r²)	total "energy" — root-sum-of-squares of singular values
Condition number κ(A)	σ₁ / σ_r	sensitivity to perturbation: κ = 10^k loses ~k digits

SVD and the Spectral Decomposition

For a symmetric positive semi-definite matrix

A

with eigenvalues

\lambda_1 \geq \cdots \geq \lambda_n \geq 0

, the spectral decomposition

A = QDQ^T

is also the SVD:

U = V = Q

and

\Sigma = D

. The singular values are the eigenvalues.

For a general symmetric matrix with some negative eigenvalues, the singular values are

|\lambda_i|

. The signs are absorbed into

U

or

V

: if

\lambda_i < 0

, one of the corresponding singular vectors is negated so that

\sigma_i = |\lambda_i| > 0

.

For non-symmetric or rectangular matrices, the eigendecomposition does not apply (it requires square matrices and may not exist even then), but the SVD always does. The SVD is the correct generalization of the spectral decomposition to the broadest possible class of matrices.

The Outer Product Form

The SVD can be written as a sum of rank-one matrices:

SVD Outer Product Form

A = \sum_{i=1}^{r} \sigma_i \, \mathbf{u}_i \mathbf{v}_i^T

Svd outer product form →

Each term

\sigma_i \mathbf{u}_i\mathbf{v}_i^T

is an

m \times n

rank-one matrix. The singular value

\sigma_i

weights its contribution. The terms are ordered by importance: the first term captures the most of

A

(in the norm sense), the second captures the most of the remainder, and so on.

Truncating this sum at

k

terms gives the best rank-

k

approximation

A_k

. The fraction of the Frobenius norm captured by the first

k

terms is

(\sigma_1^2 + \cdots + \sigma_k^2)/(\sigma_1^2 + \cdots + \sigma_r^2)

.

This outer product perspective is the basis of nearly every matrix approximation method: keep the large singular values (signal) and discard the small ones (noise or redundancy).

What the SVD Reveals

No other single factorization provides as much structural information about a matrix.

The rank: the number of nonzero singular values.

The four fundamental subspaces: orthonormal bases from the columns of

U

and

V

.

The pseudoinverse:

A^+ = V\Sigma^+ U^T

.

The best rank-

k

approximation: truncate at

k

terms.

Norms and the condition number: directly from the singular values.

The geometry of the linear map: rotation, scaling, rotation.

For symmetric matrices, the SVD reduces to the spectral decomposition. For invertible square matrices, the singular values reveal the conditioning that the determinant alone cannot see (a matrix with

\det = 1

can still be poorly conditioned). For rectangular matrices, the SVD is the only factorization that applies without modification.

The SVD is the culmination of the decomposition hierarchy — the most general, most informative, and most broadly applicable factorization in linear algebra.

The six structural quantities the SVD exposes — rank, the four fundamental subspaces, the pseudoinverse, the best rank-

k

approximation, norms and condition number, and the geometric decomposition — collect into a single reference card below.

Grouping these by which factor supplies the answer is what makes them one computation rather than seven. Three come from the singular values alone, two from the columns of the orthogonal factors, and two require reading all three factors together. Nothing below needs the matrix again once the factorization is in hand.

Download

PNGRaster · 2×PDFPrint quality

Decompositions · SVD

What the SVD reveals

Every entry comes from the same three factors. What changes is which one is being read — the singular values for size, the columns of $U$ and $V$ for structure, and all three together for geometry.

7readings

From the singular values3

1

Rank§ 3

strictly positive only

r =

count of

\sigma_i > 0

The most numerically reliable rank there is. Row reduction decides rank by comparing entries against zero, which floating point makes arbitrary; the singular values instead show a gap, and where that gap falls is a judgement the numbers themselves support.

2

Norms§ 9

read straight off the list

\|A\|_2 = \sigma_1, \quad \|A\|_F^2 = \textstyle\sum \sigma_i^2

The largest singular value is the most any unit vector is stretched. The Frobenius norm is the whole list in quadrature — so both common matrix norms are functions of the same numbers.

3

Condition number§ 9

A

of full rank

\kappa(A) = \sigma_1 / \sigma_r

The ratio of most to least stretched. A large

\kappa

means the matrix is nearly singular in some direction, and small changes to

\mathbf{b}

produce large changes to the solution — which no determinant reports, since a matrix can have

\det = 1

and be badly conditioned.

From the columns of U and V2

4

The four subspaces§ 6

orthonormal bases, all four at once

partition

U

and

V

at index

r

First

r

columns of

V

span the row space and the rest the null space; first

r

of

U

span the column space and the rest the left null space. The only method giving all four orthonormally from one computation.

5

Pseudoinverse§ 7

reciprocate the nonzero

\sigma

, leave zeros alone

A^{+} = V\Sigma^{+}U^{\mathsf{T}}

Defined for every matrix, square or not, invertible or not. When

A

is invertible it coincides with

A^{-1}

; otherwise it returns the least-squares solution of minimum norm, which is the sense in which it is the closest thing to an inverse.

From all three factors2

6

Low-rank approximation§ 8

optimal for every unitarily invariant norm

A_k = \sum_{i=1}^{k} \sigma_i \mathbf{u}_i \mathbf{v}_i^{\mathsf{T}}

Truncating the outer-product sum gives the best rank-

k

approximation there is — the Eckart–Young theorem, and it is why the SVD underlies image compression and principal component analysis. The error is exactly

\sigma_{k+1}

, so the singular values say in advance how much is lost.

7

Geometric action§ 2

every matrix, no hypotheses

A = U\Sigma V^{\mathsf{T}}

— rotate, stretch, rotate

Read right to left:

V^{\mathsf{T}}

rotates,

\Sigma

scales along axes,

U

rotates again. Every linear map is those three steps, which is the claim that makes the factorization worth having as geometry rather than only as algebra.

No other factorization answers this many questions, and none exists for every matrix. That combination is why the SVD is the one to reach for when the matrix is rectangular, rank-deficient, or simply unknown — the conditions the other decompositions require are exactly the ones it does without.

What the SVD reveals·/linear-algebra/decompositions/svdLearn Math Class

Two of these deserve emphasis because nothing else supplies them. The condition number measures how badly a system amplifies error, and it is invisible to the determinant — a matrix can have determinant one and still be nearly singular in some direction. Low-rank approximation is optimal rather than merely reasonable: truncating the sum at

k

terms gives the closest rank-

k

matrix in every unitarily invariant norm, with error exactly

\sigma_{k+1}

.

The reason all of this comes from one factorization is that the SVD asks nothing of the matrix. It need not be square, invertible, symmetric or full rank — the factorization exists regardless, which is what separates it from every other decomposition in this section and why it is the one to reach for when the matrix is unknown.

Singular Value Decomposition FAQ

Is the $U$ in the SVD the same kind of matrix as the $U$ in $LU$ ?

+

No, and the collision is worth watching. In

LU

the letter means upper triangular; in the SVD it means orthogonal. The same character signs two different contracts a couple of pages apart, and only the name of the decomposition says which is in force. Nothing about the symbol itself distinguishes them.Read more →

Is the $\Sigma$ in $U\Sigma V^{T}$ a summation sign?

+

No, it is a matrix. A summation sign carries bounds above and below and is followed by a summand, none of which appear here. A bare capital sigma sitting between two matrix letters is a factor in a product, and in the SVD it is the rectangular diagonal matrix holding the singular values with zero padding.Read more →

What is the difference between $\sigma_i$ and $\sigma(A)$ ?

+

The subscript changes the species entirely. A subscripted

\sigma_i

is one singular value, a single non-negative number, while a bare

\sigma(A)

is the spectrum, meaning the set of eigenvalues. Since singular values and eigenvalues agree only for special matrices, the two notations rarely even hold the same numbers.Read more →

Singular Value Decomposition

The Universal Matrix Factorization

What the Singular Value Decomposition Is

Diagonalization and Its Requirement

Two Bases Instead of One

What Follows and What Does Not

The Geometric Interpretation

SVD Notation

SVD Notation

Singular Values

Computing the SVD

Worked Example

Compact and Thin Forms

Three forms, one factorization

SVD and the Four Fundamental Subspaces

The Pseudoinverse

Low-Rank Approximation

SVD and Norms

SVD and the Spectral Decomposition

The Outer Product Form

What the SVD Reveals

What the SVD reveals

Singular Value Decomposition FAQ

Is the UUU in the SVD the same kind of matrix as the UUU in LULULU?

Is the Σ\SigmaΣ in UΣVTU\Sigma V^{T}UΣVT a summation sign?

What is the difference between σi\sigma_iσi​ and σ(A)\sigma(A)σ(A)?

Is the $U$ in the SVD the same kind of matrix as the $U$ in $LU$ ?

Is the $\Sigma$ in $U\Sigma V^{T}$ a summation sign?

What is the difference between $\sigma_i$ and $\sigma(A)$ ?