Projections: Formulas & Matrix Properties

Projection onto a Vector

The orthogonal projection of

\mathbf{b}

onto a nonzero vector

\mathbf{a}

is the point on the line through

\mathbf{a}

nearest to

\mathbf{b}

:

Projection onto a Vector

\text{proj}_{\mathbf{a}}\mathbf{b} = \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{a} \cdot \mathbf{a}}\,\mathbf{a}

The scalar

\hat{c} = \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{a} \cdot \mathbf{a}}

is the component of

\mathbf{b}

in the direction of

\mathbf{a}

. The projection

\hat{c}\,\mathbf{a}

lies on the line through

\mathbf{a}

, and the residual

\mathbf{b} - \hat{c}\,\mathbf{a}

is orthogonal to

\mathbf{a}

:

Projection onto a Vector

(\mathbf{b} - \hat{c}\,\mathbf{a}) \cdot \mathbf{a} = \mathbf{b} \cdot \mathbf{a} - \hat{c}(\mathbf{a} \cdot \mathbf{a}) = \mathbf{b} \cdot \mathbf{a} - \mathbf{b} \cdot \mathbf{a} = 0

Worked Example

Project

\mathbf{b} = (3, 4, 0)

onto

\mathbf{a} = (1, 1, 1)

:

Projection onto a Vector

\hat{c} = \frac{3 + 4 + 0}{1 + 1 + 1} = \frac{7}{3}, \quad \text{proj}_{\mathbf{a}}\mathbf{b} = \frac{7}{3}(1, 1, 1) = \left(\frac{7}{3}, \frac{7}{3}, \frac{7}{3}\right)

Residual:

\mathbf{b} - \text{proj}_{\mathbf{a}}\mathbf{b} = (\frac{2}{3}, \frac{5}{3}, -\frac{7}{3})

. Check:

(\frac{2}{3})(1) + (\frac{5}{3})(1) + (-\frac{7}{3})(1) = 0

.

Pitfall: Dividing by the length instead of its square

Dividing

\mathbf{a} \cdot \mathbf{b} = 7

by

\|\mathbf{a}\| = \sqrt{3}

instead of

\mathbf{a} \cdot \mathbf{a} = 3

gives

\frac{7}{\sqrt{3}}(1, 1, 1)

, of length

7

, more than

\|\mathbf{b}\| = 5

. The projection is

\frac{7}{3}(1, 1, 1)

.

The residual check is the quickest test of a computed projection. Only one multiple of

\mathbf{a}

leaves a remainder perpendicular to

\mathbf{a}

, so a slip in the coefficient shows up as a nonzero dot product. A second test is size: the projection and the residual are the two legs of a right triangle whose hypotenuse is

\mathbf{b}

, so the projection is never longer than

\mathbf{b}

.

The coefficient applied to the direction

The scalar built from the two dot products is being multiplied onto the direction vector, turning a ratio into an actual vector on that line. The projection is always a multiple of the vector being projected onto, never of the one being projected — an asymmetry the formula hides and the picture does not. Follow the whole computation on the vector projection visualizer.

Projecting onto a subspace rather than a single vector is this same step repeated once per basis direction, provided the basis is orthogonal.

Projection Notation

Notation

Projection Notation

A subscript that names the target, the hat's second job in this subject, and the letter P moonlighting for the third time.

The dot and its bars — dot product notation;

W^{\perp}

— inner product notation; the hat's first job — norm notation.

\text{proj}_{\mathbf{a}}\mathbf{b}

·

\text{proj}_{W}\mathbf{b}

The projection of b onto a; onto W

The subscript names the target — onto what — and the argument is the vector being projected. Onto a vector in Projection onto a Vector above; onto a whole subspace as the sections below build up.

CasesThe subscript's species matters:

\text{proj}_{\mathbf{a}}

lands on a line,

\text{proj}_{W}

on a subspace of any dimension — same word-operator, widening target.

Also written

P_W\mathbf{b}

— the matrix form, once The Projection Matrix below is built;

\hat{\mathbf{b}}

— the hat shorthand of the next entry.

Do not confuseA component index.

\text{proj}_{\mathbf{a}}

is not the “

\mathbf{a}

-th” anything — the subscript is a destination, joining the basis-labels of coordinate notation in the family of subscripts that name rather than count.

\hat{\mathbf{b}}

·

\hat{\mathbf{x}}

b hat — the projection; x hat — the least-squares solution

The hat's second job in linear algebra: best approximation of.

\hat{\mathbf{b}}

is the closest point to

\mathbf{b}

inside

W

— the decomposition

\mathbf{b} = \hat{\mathbf{b}} + \mathbf{z}

of The Orthogonal Decomposition below — and

\hat{\mathbf{x}}

the closest-fit solution in least squares.

CasesTwo hats now live in this subject: the unit-vector hat promises norm one; this one promises nearness. Bold-letter context usually decides —

\hat{\mathbf{u}}

tends to be unit,

\hat{\mathbf{b}}

,

\hat{\mathbf{x}}

tend to be estimates.

Also written

\mathbf{b}_{W}

or

\mathbf{b}_{\parallel}

in some texts — subscript-flavoured alternatives that free the hat for unit vectors.

Do not confuseStatistics arrived here first: the estimator hat

\hat{p}

,

\hat{\beta}

is this hat — least squares is where the two fields share one convention honestly, not a collision at all.

P = A(A^{T}A)^{-1}A^{T}

,

\quad P^2 = P

The projection matrix; applying it twice changes nothing

The letter

P

's third job on this site: change-of-basis matrix, permutation matrix, and now projection — built in The Projection Matrix below. The equation

P^2 = P

is idempotence written as notation: project twice, land in the same place.

CasesOrthogonal projections add symmetry:

P^{T} = P

. The pair

P^2 = P = P^{T}

characterizes orthogonal projection matrices — two equations serving as a definition.

Also written

P_W

with the target as subscript, matching

\text{proj}_{W}

;

\Pi

in texts that need

P

elsewhere.

Do not confuseInvertibility.

P^2 = P

makes

P

singular whenever

W \neq \mathbb{R}^n

— projection discards the

W^{\perp}

component, and discarded information is exactly what an inverse cannot recover.

All linear algebra symbols →↑ Orthogonality

The Orthogonal Decomposition

Every vector

\mathbf{b} \in \mathbb{R}^n

decomposes uniquely with respect to a subspace

W

as

Orthogonal Decomposition

\mathbf{b} = \hat{\mathbf{b}} + \mathbf{z}, \quad \hat{\mathbf{b}} \in W, \ \mathbf{z} \in W^\perp

Orthogonal decomposition →

The component

\hat{\mathbf{b}}

is the orthogonal projection of

\mathbf{b}

onto

W

, and

\mathbf{z} = \mathbf{b} - \hat{\mathbf{b}}

is the perpendicular residual.

The projection

\hat{\mathbf{b}}

is the closest point in

W

to

\mathbf{b}

. For any other vector

\mathbf{w} \in W

:

The Orthogonal Decomposition

\|\mathbf{b} - \mathbf{w}\|^2 = \|\mathbf{z}\|^2 + \|\hat{\mathbf{b}} - \mathbf{w}\|^2 \geq \|\mathbf{z}\|^2 = \|\mathbf{b} - \hat{\mathbf{b}}\|^2

Fact: Closest, yet 9% too high

The Fourier partial sums of a square wave are its projections onto more and more waves, yet beside each jump they overshoot by about 9% of the jump, however many terms are taken. Closest means least total squared error, and the spike narrows as terms are added, so its share of that error shrinks to zero.

The inequality follows from the Pythagorean theorem:

\mathbf{z}

is orthogonal to

\hat{\mathbf{b}} - \mathbf{w}

(both

\hat{\mathbf{b}}

and

\mathbf{w}

are in

W

, so their difference is in

W

, and

\mathbf{z} \in W^\perp

). The minimum distance

\|\mathbf{z}\|

is achieved uniquely at

\mathbf{w} = \hat{\mathbf{b}}

.

The argument uses nothing but the Pythagorean theorem, so it holds in any inner product space, including spaces of functions where the dot product is the integral of a product. There, projecting a function onto the span of

1, \cos x, \sin x, \dots, \cos nx, \sin nx

gives the combination of those waves with the least total squared error, and that projection is exactly the Fourier partial sum of the function.

What is left after the projection is removed

Subtracting the projection leaves a remainder orthogonal to the direction projected onto. Every vector splits this way, into a part along a subspace and a part perpendicular to it, and the split is unique. That uniqueness is what makes least squares well posed later. Check the remainder's dot product on the vector projection visualizer.

The perpendicular part is the error, and minimising it is what the whole of least squares is about.

Projection with an Orthogonal Basis

When

W = \text{Span}\{\mathbf{u}_1, \dots, \mathbf{u}_k\}

and the basis

\{\mathbf{u}_1, \dots, \mathbf{u}_k\}

is orthogonal, the projection of

\mathbf{b}

onto

W

decomposes into independent vector projections:

Projection with an Orthogonal Basis

\text{proj}_W \mathbf{b} = \frac{\mathbf{u}_1 \cdot \mathbf{b}}{\mathbf{u}_1 \cdot \mathbf{u}_1}\,\mathbf{u}_1 + \frac{\mathbf{u}_2 \cdot \mathbf{b}}{\mathbf{u}_2 \cdot \mathbf{u}_2}\,\mathbf{u}_2 + \cdots + \frac{\mathbf{u}_k \cdot \mathbf{b}}{\mathbf{u}_k \cdot \mathbf{u}_k}\,\mathbf{u}_k

Each term is the projection of

\mathbf{b}

onto one basis vector. Orthogonality prevents interference: projecting onto

\mathbf{u}_1

does not affect the component along

\mathbf{u}_2

, because

\mathbf{u}_1 \cdot \mathbf{u}_2 = 0

.

When the basis is orthonormal, the denominators are all

1

:

Projection onto Orthonormal Basis

\text{proj}_W \mathbf{b} = \sum_{i=1}^{k} (\mathbf{q}_i \cdot \mathbf{b})\,\mathbf{q}_i

Projection onto orthonormal basis →

This is the cleanest formula in all of linear algebra —

k

dot products and

k

scalar multiplications.

Projection with an Arbitrary Basis

When the basis for

W

is not orthogonal, the individual vector projection formula does not apply — projecting onto one basis vector interferes with the others. Instead, the projection requires solving a system.

If the columns of the

m \times k

matrix

A

form a basis for

W

, the projection of

\mathbf{b}

onto

W

is

Projection onto Subspace

\hat{\mathbf{b}} = A(A^TA)^{-1}A^T\mathbf{b}

Projection onto subspace →

This formula requires

A^TA

to be invertible, which holds whenever the columns of

A

are linearly independent.

Pitfall: Splitting the inverse of a product

Writing

(A^TA)^{-1} = A^{-1}(A^T)^{-1}

cancels the formula to

P = I

, but an

m \times k

matrix

A

with

k < m

has no inverse. For the columns

(1, 0, 1)

and

(0, 1, 1)

,

P = \frac{1}{3}\begin{pmatrix} 2 & -1 & 1 \\ -1 & 2 & 1 \\ 1 & 1 & 2 \end{pmatrix}

, not

I

.

The derivation comes from the orthogonality condition. The residual

\mathbf{b} - A\hat{\mathbf{x}}

must be perpendicular to every column of

A

:

A^T(\mathbf{b} - A\hat{\mathbf{x}}) = \mathbf{0}

. Solving for

\hat{\mathbf{x}}

gives

A^TA\hat{\mathbf{x}} = A^T\mathbf{b}

, so

\hat{\mathbf{x}} = (A^TA)^{-1}A^T\mathbf{b}

, and

\hat{\mathbf{b}} = A\hat{\mathbf{x}} = A(A^TA)^{-1}A^T\mathbf{b}

.

The alternative is to first orthogonalize the basis using Gram-Schmidt, then use the simpler orthogonal formula. Both approaches give the same projection.

Download

PNGRaster · 2×PDFPrint quality

Orthogonality · projection formulas

Four formulas, one operation

All four compute the same thing — the closest point of the target to $\mathbf{b}$ . They differ only in what is assumed about the basis, and the formula gets simpler as the assumption gets stronger.

4cases

Onto a single vector1

A line§ 1the target is

\operatorname{Span}\{\mathbf{a}\}

formula

\dfrac{\mathbf{b} \cdot \mathbf{a}}{\mathbf{a} \cdot \mathbf{a}}\,\mathbf{a}

requires

\mathbf{a} \neq \mathbf{0}

cost per projectiontwo dot products

The base case, and the one every formula below is built from. The denominator is what normalisation removes — for a unit vector it is

1

and disappears.

Onto a subspace — basis assumed orthogonal2

Orthogonal basis§ 3

\mathbf{u}_i \cdot \mathbf{u}_j = 0

for

i \neq j

formula

\sum \dfrac{\mathbf{b} \cdot \mathbf{u}_i}{\mathbf{u}_i \cdot \mathbf{u}_i}\,\mathbf{u}_i

requirespairwise orthogonal, nonzero

cost per projection

2k

dot products

A sum of independent line projections. Orthogonality is what makes the terms independent — each coefficient can be computed without knowing the others, and adding a basis vector does not disturb them.

Orthonormal basis§ 3orthogonal and

\|\mathbf{q}_i\| = 1

formula

\sum (\mathbf{b} \cdot \mathbf{q}_i)\,\mathbf{q}_i

requirespairwise orthogonal, unit length

cost per projection

k

dot products

The denominators vanish because each is

1

. This is the form worth arranging for — see orthonormal operations, where every operation simplifies the same way.

Onto a subspace — nothing assumed1

Arbitrary basis§ 4columns of

A

span the target

formula

A(A^{\mathsf{T}}A)^{-1}A^{\mathsf{T}}\mathbf{b}

requirescolumns of

A

independent

cost per projectiona

k \times k

inverse

The general case, and the expensive one.

(A^{\mathsf{T}}A)^{-1}

exists exactly when the columns are independent — and when they happen to be orthonormal,

A^{\mathsf{T}}A = I

and this collapses to the row above.

Reading upward is the argument for Gram–Schmidt. The general formula needs a matrix inverse; orthogonality removes the coupling between coefficients; normalisation removes the denominators. Each assumption bought by preprocessing is a computation avoided on every projection afterwards.

Four formulas, one operation·/linear-algebra/orthogonality/projectionsLearn Math Class

Ordered this way the four are a single formula losing assumptions. Start at the bottom with an arbitrary basis and the projection needs a

k \times k

matrix inverse. Assume the basis orthogonal and the inverse disappears, because

A^{\mathsf{T}}A

becomes diagonal and the coefficients stop depending on each other. Assume it orthonormal as well and the denominators go too, since each is one.

The Projection Matrix

The matrix

P = A(A^TA)^{-1}A^T

maps any vector

\mathbf{b}

to its projection onto

\text{Col}(A)

:

\hat{\mathbf{b}} = P\mathbf{b}

.

When the basis is orthonormal (

A = Q

with

Q^TQ = I

), the formula simplifies:

Orthonormal Columns Projection

P = QQ^T

Orthonormal columns projection →

Pitfall: Taking QQ transpose to be the identity

Concluding

QQ^T = I

from

Q^TQ = I

fails for a non-square

Q

: with the single column

\frac{1}{\sqrt{2}}(1, 1, 0)

,

Q^TQ = (1)

, but

QQ^T = \frac{1}{2}\begin{pmatrix} 1 & 1 & 0 \\ 1 & 1 & 0 \\ 0 & 0 & 0 \end{pmatrix}

sends

(1, 0, 0)

to

(\frac{1}{2}, \frac{1}{2}, 0)

.

The projection matrix satisfies two algebraic conditions. It is symmetric:

P^T = P

. And it is idempotent:

P^2 = P

— projecting twice gives the same result as projecting once, because vectors already in

W

are fixed by

P

.

The complementary matrix

I - P

projects onto the orthogonal complement

W^\perp

:

Complementary Projection

I - P \text{ projects onto } W^\perp

Complementary projection →

It satisfies

(I - P)^T = I - P

and

(I - P)^2 = I - P

, and for every

\mathbf{b}

:

P\mathbf{b} + (I - P)\mathbf{b} = \mathbf{b}

, decomposing

\mathbf{b}

into its

W

-component and its

W^\perp

-component.

The eigenvalues of

P

are

0

and

1

: vectors in

W

map to themselves (eigenvalue

1

) and vectors in

W^\perp

map to zero (eigenvalue

0

).

RULE

The rank of

P

equals the trace of

P

, which equals

\dim(W)

.

Property of P	Statement	Why it holds / what it means
Idempotent	P² = P	projecting a second time changes nothing; vectors already in W are fixed
Symmetric	Pᵀ = P	makes the projection orthogonal (residual ⊥ W); distinguishes from oblique projections, which satisfy P² = P but Pᵀ ≠ P
Complementary projection	I − P projects onto W^⊥	I − P is also symmetric and idempotent
Decomposition of every b	b = Pb + (I − P) b	unique split into a W-component and a W^⊥-component
Eigenvalues	only 0 and 1	vectors in W are eigenvectors for λ = 1; vectors in W^⊥ for λ = 0
Rank	rank(P) = dim(W)	number of independent directions preserved
Trace	tr(P) = dim(W)	trace = sum of eigenvalues = number of eigenvalues equal to 1
Orthonormal basis simplification	P = QQᵀ when A = Q has Qᵀ Q = I	the (AᵀA)⁻¹ factor disappears because QᵀQ = I

The whole plane collapsed onto a line

Every point has been dropped perpendicularly onto the line, and points already on it have not moved at all. That is why applying the matrix twice changes nothing the second time: idempotence is a geometric fact here, not an algebraic coincidence. Move the line and watch the whole plane follow on the 2D projection explorer.

A projection matrix is therefore recognisable by squaring to itself, without any need to know which subspace it projects onto.

Properties of Orthogonal Projections

Orthogonal projections are characterized by two properties acting together:

Projection Matrix Properties

P^2 = P, \quad P^T = P

Projection matrix properties →

Idempotence (

P^2 = P

): once a vector has been projected, projecting again changes nothing. Every vector in

W

is a fixed point of

P

. This distinguishes projections from other linear transformations — most transformations continue to change vectors on repeated application.

Fact: An evening shadow outgrows its pole

With the sun lower than

45°

, a pole casts a shadow longer than itself: sunlight projects onto the ground along a slanted direction, an oblique projection with

P^2 = P

but

P^T \neq P

. An orthogonal projection never lengthens a vector, and a projection that never lengthens one is always orthogonal.

Symmetry (

P^T = P

): the projection is self-adjoint with respect to the dot product. This means

P\mathbf{u} \cdot \mathbf{v} = \mathbf{u} \cdot P\mathbf{v}

for all

\mathbf{u}, \mathbf{v}

. The symmetry condition is what makes the projection orthogonal rather than oblique — it ensures the residual is perpendicular to

W

, not merely non-parallel.

A matrix satisfying

P^2 = P

and

P^T = P

is an orthogonal projection. A matrix satisfying

P^2 = P

but

P^T \neq P

is an oblique projection — it projects onto the same subspace but along a different direction, not the perpendicular one.

The error

\|\mathbf{b} - P\mathbf{b}\|

is the distance from

\mathbf{b}

to

W

. It is the smallest possible value of

\|\mathbf{b} - \mathbf{w}\|

over all

\mathbf{w} \in W

.

Download

PNGRaster · 2×PDFPrint quality

Orthogonality · projections

Properties of orthogonal projections

Two conditions define the whole object, and everything below them is a consequence. Idempotence alone gives a projection; symmetry is what makes it the perpendicular one.

07properties

Unconditional4

i

Idempotence§ 6

P^2 = P

Projecting twice changes nothing — once a vector is in W it stays put, so every vector of W is…

Unconditional

Requires: defining condition

Projecting twice changes nothing — once a vector is in $W$ it stays put, so every vector of $W$ is a fixed point of $P$ . This is unusual among linear transformations, most of which keep changing a vector on repeated application.

ii

Symmetry§ 6

P^{\mathsf{T}} = P

Self-adjoint with respect to the dot product, meaning Pu · v = u · Pv for every pair.

Unconditional

Requires: defining condition

Self-adjoint with respect to the dot product, meaning $P\mathbf{u} \cdot \mathbf{v} = \mathbf{u} \cdot P\mathbf{v}$ for every pair. This is the condition that makes the projection orthogonal rather than merely a projection — it forces the residual to be perpendicular to $W$ , not just outside it.

v

Complementary projection§ 2

I - P

projects onto

W^{\perp}

The residual is itself a projection, onto the orthogonal complement.

Unconditional

Requires: $P$ orthogonal

The residual is itself a projection, onto the orthogonal complement. Check: $(I-P)^2 = I - 2P + P^2 = I - P$ by idempotence, and $(I-P)^{\mathsf{T}} = I - P$ by symmetry — so it satisfies both defining conditions.

vii

Eigenvalues§ 5

\lambda \in \{0, 1\}

From Pv = λv and idempotence, λ^2 = λ, so λ is 0 or 1.

Unconditional

Requires: follows from $P^2 = P$

From $P\mathbf{v} = \lambda\mathbf{v}$ and idempotence, $\lambda^2 = \lambda$ , so $\lambda$ is $0$ or $1$ . The eigenvectors for $1$ span $W$ , those for $0$ span $W^\perp$ , and $\operatorname{rank}(P) = \operatorname{tr}(P) = \dim W$ — the trace counts the ones. See spectral properties.

Conditional2

iv

Orthogonal decomposition§ 2

\mathbf{b} = P\mathbf{b} + (\mathbf{b} - P\mathbf{b})

Every vector splits uniquely into a part in W and a part perpendicular to it.

Conditional

Holds when: $P$ orthogonal — both conditions

Every vector splits uniquely into a part in $W$ and a part perpendicular to it. Uniqueness is what makes this a decomposition rather than merely one way of writing $\mathbf{b}$ , and it is the reason $W$ and $W^\perp$ intersect only at the origin.

vi

Minimal distance§ 7

\|\mathbf{b} - P\mathbf{b}\| \leq \|\mathbf{b} - \mathbf{w}\|

for all

\mathbf{w} \in W

The projection is the closest point of W to b, and the only one achieving the minimum.

Conditional

Holds when: equality iff $\mathbf{w} = P\mathbf{b}$

The projection is the closest point of $W$ to $\mathbf{b}$ , and the only one achieving the minimum. This is the entire content of least squares — the best approximation is a projection, and the perpendicularity of the residual is why no other point can do better.

Fails or undefined1

iii

Oblique projection§ 6

P^2 = P

but

P^{\mathsf{T}} \neq P

Still a projection, and still onto the same subspace — but along a slanted direction rather…

Not orthogonal

Fails when: symmetry is dropped
Common error: treating idempotence alone as the definition of a projection onto $W$

Still a projection, and still onto the same subspace — but along a slanted direction rather than the perpendicular one. The two conditions are independent, and this entry exists to show that the second is not implied by the first.

WitnessP = [[1, 1], [0, 0]]
P² = P, so it projects — onto the x-axis
Pᵀ ≠ P, so it projects along the line y = −x, not perpendicular

Unconditional

Conditional

Fails or undefined

Idempotence§ 6Unconditional

P^2 = P

Requires: defining condition

Projecting twice changes nothing — once a vector is in

W

it stays put, so every vector of

W

is a fixed point of

P

. This is unusual among linear transformations, most of which keep changing a vector on repeated application.

Read the full section

Symmetry§ 6Unconditional

P^{\mathsf{T}} = P

Requires: defining condition

Self-adjoint with respect to the dot product, meaning

P\mathbf{u} \cdot \mathbf{v} = \mathbf{u} \cdot P\mathbf{v}

for every pair. This is the condition that makes the projection orthogonal rather than merely a projection — it forces the residual to be perpendicular to

W

, not just outside it.

Read the full section

Oblique projection§ 6Not orthogonal

P^2 = P

but

P^{\mathsf{T}} \neq P

Fails when: symmetry is dropped
Common error: treating idempotence alone as the definition of a projection onto $W$

Still a projection, and still onto the same subspace — but along a slanted direction rather than the perpendicular one. The two conditions are independent, and this entry exists to show that the second is not implied by the first.

WitnessP = [[1, 1], [0, 0]]
P² = P, so it projects — onto the x-axis
Pᵀ ≠ P, so it projects along the line y = −x, not perpendicular

Read the full section

Orthogonal decomposition§ 2Conditional

\mathbf{b} = P\mathbf{b} + (\mathbf{b} - P\mathbf{b})

Holds when: $P$ orthogonal — both conditions

Every vector splits uniquely into a part in

W

and a part perpendicular to it. Uniqueness is what makes this a decomposition rather than merely one way of writing

\mathbf{b}

, and it is the reason

W

and

W^\perp

intersect only at the origin.

Read the full section

Complementary projection§ 2Unconditional

I - P

projects onto

W^{\perp}

Requires: $P$ orthogonal

The residual is itself a projection, onto the orthogonal complement. Check:

(I-P)^2 = I - 2P + P^2 = I - P

by idempotence, and

(I-P)^{\mathsf{T}} = I - P

by symmetry — so it satisfies both defining conditions.

Read the full section

Minimal distance§ 7Conditional

\|\mathbf{b} - P\mathbf{b}\| \leq \|\mathbf{b} - \mathbf{w}\|

for all

\mathbf{w} \in W

Holds when: equality iff $\mathbf{w} = P\mathbf{b}$

The projection is the closest point of

W

to

\mathbf{b}

, and the only one achieving the minimum. This is the entire content of least squares — the best approximation is a projection, and the perpendicularity of the residual is why no other point can do better.

Read the full section

Eigenvalues§ 5Unconditional

\lambda \in \{0, 1\}

Requires: follows from $P^2 = P$

From

P\mathbf{v} = \lambda\mathbf{v}

and idempotence,

\lambda^2 = \lambda

, so

\lambda

is

0

or

1

. The eigenvectors for

1

span

W

, those for

0

span

W^\perp

, and

\operatorname{rank}(P) = \operatorname{tr}(P) = \dim W

— the trace counts the ones. See spectral properties.

Read the full section

Drop symmetry and

P

still projects onto

W

, but along some other direction — an oblique projection. The residual is then no longer perpendicular, the distance is no longer minimal, and least squares loses its justification entirely. That is how much rests on the second condition.

Properties of orthogonal projections·/linear-algebra/orthogonality/projectionsLearn Math Class

Squaring to itself makes a matrix a projection; symmetry is what makes it drop straight down, onto the closest point.

Projection and Least Squares

When the system

A\mathbf{x} = \mathbf{b}

has no solution — when

\mathbf{b}

is not in the column space of

A

— the least-squares solution

\hat{\mathbf{x}}

produces the projection of

\mathbf{b}

onto

\text{Col}(A)

:

Projection and Least Squares

A\hat{\mathbf{x}} = \hat{\mathbf{b}} = P\mathbf{b}

The least-squares solution does not solve

A\mathbf{x} = \mathbf{b}

. It solves

A\mathbf{x} = \hat{\mathbf{b}}

, where

\hat{\mathbf{b}}

is the closest reachable vector to

\mathbf{b}

.

RULE

The residual

\mathbf{r} = \mathbf{b} - A\hat{\mathbf{x}}

lies in

\text{Col}(A)^\perp = \text{Null}(A^T)

— it is orthogonal to every column of

A

.

The condition

A^T\mathbf{r} = \mathbf{0}

is exactly the normal equation

A^TA\hat{\mathbf{x}} = A^T\mathbf{b}

.

Every least-squares problem is a projection problem. Solving least squares means projecting the target

\mathbf{b}

onto the column space and finding the input

\hat{\mathbf{x}}

that produces the projected output.

Worked Example: Full Projection Computation

Project

\mathbf{b} = (1, 2, 3)

onto the subspace

W = \text{Span}\{(1, 0, 1), (0, 1, 1)\}

in

\mathbb{R}^3

.

The basis is not orthogonal:

(1, 0, 1) \cdot (0, 1, 1) = 0 + 0 + 1 = 1 \neq 0

. Use the general formula. Set

A = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 1 & 1 \end{pmatrix}

.