Chapter 11

Gram-Schmidt Process

00 · Symbol Glossary

$\text{proj}_{\mathbf{u}}\mathbf{v}$proj u v — projection of v onto u

The component of $\mathbf{v}$ that lies in the direction of $\mathbf{u}$ . Computed as $\frac{\langle\mathbf{v},\mathbf{u}\rangle}{\langle\mathbf{u},\mathbf{u}\rangle}\mathbf{u}$ . The projection is a vector, not a scalar — it lives in $\mathbb{R}^n$ in the same direction as $\mathbf{u}$ . Subtracting it from $\mathbf{v}$ removes the $\mathbf{u}$ -direction entirely, leaving a vector orthogonal to $\mathbf{u}$ .

$Q, R$Q R — QR factors

The QR decomposition factors any matrix $A$ with linearly independent columns as $A = QR$ , where $Q$ has orthonormal columns and $R$ is upper triangular. The columns of $Q$ are produced by Gram-Schmidt applied to the columns of $A$ ; the entries of $R$ record the projection coefficients at each step.

$\mathbf{e}_k^*$e star k — orthonormalised vector

The $k$ -th orthonormal basis vector produced by Gram-Schmidt. First compute the orthogonalised vector $\mathbf{u}_k$ (by subtracting all projections), then normalise: $\mathbf{e}_k^* = \mathbf{u}_k / \|\mathbf{u}_k\|$ . Not to be confused with the standard basis vectors $\mathbf{e}_k$ .

01 · The Problem Gram-Schmidt Solves

Given any linearly independent set $\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}$ , the goal is to produce an orthonormal set $\{\mathbf{q}_1, \ldots, \mathbf{q}_k\}$ that spans the same space. The two sets are related: for every $m \leq k$ , $\text{span}\{\mathbf{v}_1, \ldots, \mathbf{v}_m\} = \text{span}\{\mathbf{q}_1, \ldots, \mathbf{q}_m\}$ .

The idea is geometric: take $\mathbf{v}_2$ and subtract the part of it that points in the $\mathbf{v}_1$ direction. What remains is orthogonal to $\mathbf{v}_1$ . Normalise it. Repeat for each subsequent vector, subtracting every previously found direction.

Definition — Orthogonal Projection Onto a Vector

The projection of $\mathbf{v}$ onto $\mathbf{u}$ is:

\text{proj}_{\mathbf{u}}\mathbf{v} = \frac{\langle \mathbf{v}, \mathbf{u} \rangle}{\langle \mathbf{u}, \mathbf{u} \rangle}\,\mathbf{u}

The scalar $\frac{\langle\mathbf{v},\mathbf{u}\rangle}{\langle\mathbf{u},\mathbf{u}\rangle}$ is the projection coefficient — how much of $\mathbf{u}$ is contained in $\mathbf{v}$ . The vector $\mathbf{v} - \text{proj}_\mathbf{u}\mathbf{v}$ is orthogonal to $\mathbf{u}$ , because $\langle \mathbf{v} - \text{proj}_\mathbf{u}\mathbf{v},\, \mathbf{u} \rangle = \langle\mathbf{v},\mathbf{u}\rangle - \frac{\langle\mathbf{v},\mathbf{u}\rangle}{\|\mathbf{u}\|^2}\|\mathbf{u}\|^2 = 0$ .

✓ Example — Projection Removes a Direction

$\mathbf{v} = \begin{pmatrix}3\\2\end{pmatrix}$ , $\mathbf{u} = \begin{pmatrix}2\\0\end{pmatrix}$ .

$\text{proj}_\mathbf{u}\mathbf{v} = \frac{3(2)+2(0)}{4+0}\begin{pmatrix}2\\0\end{pmatrix} = \frac{6}{4}\begin{pmatrix}2\\0\end{pmatrix} = \begin{pmatrix}3\\0\end{pmatrix}$ .

Remainder: $\mathbf{v} - \text{proj}_\mathbf{u}\mathbf{v} = \begin{pmatrix}3\\2\end{pmatrix}-\begin{pmatrix}3\\0\end{pmatrix} = \begin{pmatrix}0\\2\end{pmatrix}$ .

Check: $\begin{pmatrix}0\\2\end{pmatrix}\cdot\begin{pmatrix}2\\0\end{pmatrix} = 0$ ✓ — the remainder is orthogonal to $\mathbf{u}$ , pointing purely in the $y$ -direction.

02 · The Gram-Schmidt Algorithm

Definition — Gram-Schmidt Process

Given linearly independent $\{\mathbf{v}_1, \ldots, \mathbf{v}_k\} \subset \mathbb{R}^n$ , compute orthonormal $\{\mathbf{q}_1, \ldots, \mathbf{q}_k\}$ by:

Step 1: $\mathbf{u}_1 = \mathbf{v}_1$ , $\quad\mathbf{q}_1 = \mathbf{u}_1/\|\mathbf{u}_1\|$ .

Step $j$ (for $j = 2, \ldots, k$ ):

\mathbf{u}_j = \mathbf{v}_j - \sum_{i=1}^{j-1} \langle \mathbf{v}_j, \mathbf{q}_i \rangle\, \mathbf{q}_i, \qquad \mathbf{q}_j = \frac{\mathbf{u}_j}{\|\mathbf{u}_j\|}

Each $\mathbf{u}_j$ is $\mathbf{v}_j$ with all previously found directions removed. $\mathbf{u}_j \neq \mathbf{0}$ because the original set was linearly independent — $\mathbf{v}_j$ cannot be a linear combination of $\mathbf{v}_1, \ldots, \mathbf{v}_{j-1}$ .

Step-by-step — Gram-Schmidt on $\mathbf{v}_1=\begin{pmatrix}1\\1\\0\end{pmatrix}$, $\mathbf{v}_2=\begin{pmatrix}1\\0\\1\end{pmatrix}$, $\mathbf{v}_3=\begin{pmatrix}0\\1\\1\end{pmatrix}$

Normalise $\mathbf{v}_1$ : $\|\mathbf{v}_1\| = \sqrt{1+1+0} = \sqrt{2}$ .

\mathbf{q}_1 = \frac{1}{\sqrt{2}}\begin{pmatrix}1\\1\\0\end{pmatrix}

Project $\mathbf{v}_2$ onto $\mathbf{q}_1$ : $\langle\mathbf{v}_2,\mathbf{q}_1\rangle = \frac{1}{\sqrt{2}}(1\cdot1 + 0\cdot1 + 1\cdot0) = \frac{1}{\sqrt{2}}$ .

Projection: $\frac{1}{\sqrt{2}} \cdot \mathbf{q}_1 = \frac{1}{\sqrt{2}}\cdot\frac{1}{\sqrt{2}}\begin{pmatrix}1\\1\\0\end{pmatrix} = \frac{1}{2}\begin{pmatrix}1\\1\\0\end{pmatrix}$ .

Subtract projection from $\mathbf{v}_2$ : remove the $\mathbf{q}_1$ component.

\mathbf{u}_2 = \begin{pmatrix}1\\0\\1\end{pmatrix} - \frac{1}{2}\begin{pmatrix}1\\1\\0\end{pmatrix} = \begin{pmatrix}1/2\\-1/2\\1\end{pmatrix}

The $\frac{1}{2}$ entries come from $1 - \frac{1}{2}=\frac{1}{2}$ and $0 - \frac{1}{2}=-\frac{1}{2}$ .

Check: $\mathbf{u}_2\cdot\mathbf{q}_1 = \frac{1}{\sqrt{2}}(\frac{1}{2}\cdot1+(-\frac{1}{2})\cdot1+1\cdot0) = 0$ ✓.

Normalise $\mathbf{u}_2$ : $\|\mathbf{u}_2\| = \sqrt{\frac{1}{4}+\frac{1}{4}+1} = \sqrt{\frac{3}{2}} = \frac{\sqrt{6}}{2}$ .

\mathbf{q}_2 = \frac{2}{\sqrt{6}}\begin{pmatrix}1/2\\-1/2\\1\end{pmatrix} = \frac{1}{\sqrt{6}}\begin{pmatrix}1\\-1\\2\end{pmatrix}

Multiplied numerator and denominator by 2: $\frac{2}{\sqrt{6}}\cdot\frac{1}{2}=\frac{1}{\sqrt{6}}$ and $\frac{2}{\sqrt{6}}\cdot1=\frac{2}{\sqrt{6}}$ .

Project $\mathbf{v}_3$ onto $\mathbf{q}_1$ and $\mathbf{q}_2$ :

$\langle\mathbf{v}_3,\mathbf{q}_1\rangle = \frac{1}{\sqrt{2}}(0+1+0) = \frac{1}{\sqrt{2}}$ .

$\langle\mathbf{v}_3,\mathbf{q}_2\rangle = \frac{1}{\sqrt{6}}(0\cdot1+1\cdot(-1)+1\cdot2) = \frac{1}{\sqrt{6}}$ .

Subtract both projections from $\mathbf{v}_3$ :

$\mathbf{u}_3 = \begin{pmatrix}0\\1\\1\end{pmatrix} - \frac{1}{\sqrt{2}}\cdot\frac{1}{\sqrt{2}}\begin{pmatrix}1\\1\\0\end{pmatrix} - \frac{1}{\sqrt{6}}\cdot\frac{1}{\sqrt{6}}\begin{pmatrix}1\\-1\\2\end{pmatrix}$

$= \begin{pmatrix}0\\1\\1\end{pmatrix} - \frac{1}{2}\begin{pmatrix}1\\1\\0\end{pmatrix} - \frac{1}{6}\begin{pmatrix}1\\-1\\2\end{pmatrix} = \begin{pmatrix}0-\frac{1}{2}-\frac{1}{6}\\1-\frac{1}{2}+\frac{1}{6}\\1-0-\frac{2}{6}\end{pmatrix} = \begin{pmatrix}-\frac{2}{3}\\\frac{2}{3}\\\frac{2}{3}\end{pmatrix}$

$\|\mathbf{u}_3\| = \sqrt{\frac{4}{9}+\frac{4}{9}+\frac{4}{9}} = \sqrt{\frac{4}{3}} = \frac{2}{\sqrt{3}}$ .

\mathbf{q}_3 = \frac{\sqrt{3}}{2}\cdot\frac{2}{3}\begin{pmatrix}-1\\1\\1\end{pmatrix} = \frac{1}{\sqrt{3}}\begin{pmatrix}-1\\1\\1\end{pmatrix}

Verify orthonormality of result: $\{\mathbf{q}_1,\mathbf{q}_2,\mathbf{q}_3\}$ are mutually orthogonal unit vectors spanning $\mathbb{R}^3$ ✓. The three original vectors and the three orthonormal vectors span the same space — any $\mathbf{v}\in\mathbb{R}^3$ is a combination of either set.

❌ What Breaks — Linear Dependence Kills Gram-Schmidt

If $\mathbf{v}_2 = 2\mathbf{v}_1$ , then $\mathbf{u}_2 = \mathbf{v}_2 - \langle\mathbf{v}_2,\mathbf{q}_1\rangle\mathbf{q}_1 = 2\mathbf{v}_1 - 2\|\mathbf{v}_1\|\cdot\frac{\mathbf{v}_1}{\|\mathbf{v}_1\|} = \mathbf{0}$ . Dividing by $\|\mathbf{u}_2\|=0$ is undefined. The algorithm collapses because $\mathbf{v}_2$ is entirely in the span of $\mathbf{v}_1$ — it adds no new direction to orthogonalise. Gram-Schmidt only works on linearly independent input.

03 · QR Decomposition

Gram-Schmidt produces a factorisation of the original matrix.

Definition — QR Decomposition

If $A \in \mathbb{R}^{m \times n}$ has linearly independent columns, then:

A = QR

where $Q \in \mathbb{R}^{m \times n}$ has orthonormal columns ( $Q^TQ = I_n$ ) and $R \in \mathbb{R}^{n \times n}$ is upper triangular with positive diagonal entries. The columns of $Q$ are the Gram-Schmidt orthonormal vectors; the entries of $R$ are the projection coefficients.

The connection: the $j$ -th column of $A$ is $\mathbf{v}_j$ . Gram-Schmidt gives $\mathbf{v}_j = r_{1j}\mathbf{q}_1 + r_{2j}\mathbf{q}_2 + \cdots + r_{jj}\mathbf{q}_j$ , so $R_{ij} = \langle\mathbf{v}_j,\mathbf{q}_i\rangle$ for $i < j$ and $R_{jj} = \|\mathbf{u}_j\|$ . Entries above the diagonal are projection coefficients; the diagonal is the norm of each orthogonalised vector; below the diagonal is zero.

Step-by-step — QR decomposition of $A=\begin{pmatrix}1&1\\1&0\\0&1\end{pmatrix}$

Run Gram-Schmidt on columns: $\mathbf{v}_1=\begin{pmatrix}1\\1\\0\end{pmatrix}$ , $\mathbf{v}_2=\begin{pmatrix}1\\0\\1\end{pmatrix}$ .

$\|\mathbf{v}_1\|=\sqrt{2}$ , so $\mathbf{q}_1 = \frac{1}{\sqrt{2}}\begin{pmatrix}1\\1\\0\end{pmatrix}$ .

Orthogonalise $\mathbf{v}_2$ : $\langle\mathbf{v}_2,\mathbf{q}_1\rangle = \frac{1}{\sqrt{2}}(1+0+0)=\frac{1}{\sqrt{2}}$ .

$\mathbf{u}_2 = \begin{pmatrix}1\\0\\1\end{pmatrix}-\frac{1}{\sqrt{2}}\cdot\frac{1}{\sqrt{2}}\begin{pmatrix}1\\1\\0\end{pmatrix} = \begin{pmatrix}1-\frac{1}{2}\\-\frac{1}{2}\\1\end{pmatrix} = \begin{pmatrix}\frac{1}{2}\\-\frac{1}{2}\\1\end{pmatrix}$ .

$\|\mathbf{u}_2\|=\sqrt{\frac{1}{4}+\frac{1}{4}+1}=\sqrt{\frac{6}{4}}=\frac{\sqrt{6}}{2}$ , so $\mathbf{q}_2=\frac{1}{\sqrt{6}}\begin{pmatrix}1\\-1\\2\end{pmatrix}$ .

Read off $R$ : $r_{11}=\|\mathbf{v}_1\|=\sqrt{2}$ ; $r_{12}=\langle\mathbf{v}_2,\mathbf{q}_1\rangle=\frac{1}{\sqrt{2}}$ ; $r_{22}=\|\mathbf{u}_2\|=\frac{\sqrt{6}}{2}$ .

Q = \begin{pmatrix}\frac{1}{\sqrt{2}}&\frac{1}{\sqrt{6}}\\\frac{1}{\sqrt{2}}&\frac{-1}{\sqrt{6}}\\0&\frac{2}{\sqrt{6}}\end{pmatrix}, \quad R = \begin{pmatrix}\sqrt{2}&\frac{1}{\sqrt{2}}\\0&\frac{\sqrt{6}}{2}\end{pmatrix}

Verify $A=QR$ : $QR$ should recover the original columns. Column 1: $\sqrt{2}\mathbf{q}_1 = \begin{pmatrix}1\\1\\0\end{pmatrix}$ ✓. Column 2: $\frac{1}{\sqrt{2}}\mathbf{q}_1+\frac{\sqrt{6}}{2}\mathbf{q}_2 = \frac{1}{2}\begin{pmatrix}1\\1\\0\end{pmatrix}+\frac{1}{2}\begin{pmatrix}1\\-1\\2\end{pmatrix}=\begin{pmatrix}1\\0\\1\end{pmatrix}$ ✓.

04 · Quant Application — Orthogonal Factors in Return Models

In factor models for asset returns, you want a set of explanatory factors that are uncorrelated. If the raw factor candidates $F_1, F_2, \ldots, F_k$ are correlated, Gram-Schmidt produces an orthogonalised factor set $G_1, G_2, \ldots, G_k$ spanning the same space but with $\text{Cov}(G_i, G_j) = 0$ for $i \neq j$ .

This matters for attribution: with correlated factors, the individual contributions of each factor to a regression overlap. Orthogonalised factors each contribute uniquely — their variance contributions are additive.

QR also appears in numerical linear algebra: least squares problems $\min \|A\mathbf{x}-\mathbf{b}\|$ are solved stably via $A=QR$ , giving $\mathbf{x} = R^{-1}Q^T\mathbf{b}$ — no need to form $A^TA$ , which can be ill-conditioned (Chapter 12).

05 · Practice Exercises

EXERCISE 11.1

Apply the projection formula: $\text{proj}_\mathbf{u}\mathbf{v} = \frac{\mathbf{v}\cdot\mathbf{u}}{\mathbf{u}\cdot\mathbf{u}}\mathbf{u}$ . Then subtract to get the component orthogonal to $\mathbf{u}$ .

$\mathbf{u}=\begin{pmatrix}3\\4\end{pmatrix}$ , $\mathbf{v}=\begin{pmatrix}5\\1\end{pmatrix}$ .

$\mathbf{u}\cdot\mathbf{u} = 9+16=25$ . $\mathbf{v}\cdot\mathbf{u} = 15+4=19$ .

$\text{proj}_\mathbf{u}\mathbf{v} = \frac{19}{25}\begin{pmatrix}3\\4\end{pmatrix} = \begin{pmatrix}57/25\\76/25\end{pmatrix}$ .

Orthogonal component: $\mathbf{v}-\text{proj}_\mathbf{u}\mathbf{v} = \begin{pmatrix}5-57/25\\1-76/25\end{pmatrix} = \begin{pmatrix}68/25\\-51/25\end{pmatrix}$ .

Verify orthogonality: $\begin{pmatrix}68/25\\-51/25\end{pmatrix}\cdot\begin{pmatrix}3\\4\end{pmatrix} = \frac{1}{25}(204-204)=0$ ✓.

Compute $\text{proj}_\mathbf{u}\mathbf{v}$ and $\mathbf{v}-\text{proj}_\mathbf{u}\mathbf{v}$ for $\mathbf{u}=\begin{pmatrix}3\\4\end{pmatrix}$ , $\mathbf{v}=\begin{pmatrix}5\\1\end{pmatrix}$ . Verify that the remainder is orthogonal to $\mathbf{u}$ .

EXERCISE 11.2

Step 1: normalise $\mathbf{v}_1$ to get $\mathbf{q}_1$ . Step 2: subtract the $\mathbf{q}_1$ -projection from $\mathbf{v}_2$ , then normalise. Check that $\mathbf{q}_1\cdot\mathbf{q}_2=0$ .

$\mathbf{v}_1=\begin{pmatrix}1\\1\end{pmatrix}$ , $\mathbf{v}_2=\begin{pmatrix}2\\0\end{pmatrix}$ .

$\mathbf{q}_1 = \frac{1}{\sqrt{2}}\begin{pmatrix}1\\1\end{pmatrix}$ .

$\langle\mathbf{v}_2,\mathbf{q}_1\rangle = \frac{1}{\sqrt{2}}(2+0)=\sqrt{2}$ .

$\mathbf{u}_2 = \begin{pmatrix}2\\0\end{pmatrix} - \sqrt{2}\cdot\frac{1}{\sqrt{2}}\begin{pmatrix}1\\1\end{pmatrix} = \begin{pmatrix}2\\0\end{pmatrix}-\begin{pmatrix}1\\1\end{pmatrix}=\begin{pmatrix}1\\-1\end{pmatrix}$ .

$\|\mathbf{u}_2\|=\sqrt{1+1}=\sqrt{2}$ , so $\mathbf{q}_2=\frac{1}{\sqrt{2}}\begin{pmatrix}1\\-1\end{pmatrix}$ .

Verify: $\mathbf{q}_1\cdot\mathbf{q}_2=\frac{1}{2}(1\cdot1+1\cdot(-1))=0$ ✓. $\|\mathbf{q}_1\|=\|\mathbf{q}_2\|=1$ ✓.

$Q=\frac{1}{\sqrt{2}}\begin{pmatrix}1&1\\1&-1\end{pmatrix}$ , $R=\begin{pmatrix}\sqrt{2}&\sqrt{2}\\0&\sqrt{2}\end{pmatrix}$ (since $r_{11}=\|\mathbf{v}_1\|=\sqrt{2}$ , $r_{12}=\langle\mathbf{v}_2,\mathbf{q}_1\rangle=\sqrt{2}$ , $r_{22}=\|\mathbf{u}_2\|=\sqrt{2}$ ).

Apply Gram-Schmidt to $\mathbf{v}_1=\begin{pmatrix}1\\1\end{pmatrix}$ , $\mathbf{v}_2=\begin{pmatrix}2\\0\end{pmatrix}$ to produce an orthonormal basis $\{\mathbf{q}_1,\mathbf{q}_2\}$ . Write the resulting QR decomposition of $A=\begin{pmatrix}1&2\\1&0\end{pmatrix}$ .

EXERCISE 11.3

Run the full Gram-Schmidt: normalise $\mathbf{v}_1$ ; orthogonalise $\mathbf{v}_2$ against $\mathbf{q}_1$ ; orthogonalise $\mathbf{v}_3$ against both $\mathbf{q}_1$ and $\mathbf{q}_2$ . Normalise each step.

$\mathbf{v}_1=\begin{pmatrix}1\\0\\0\end{pmatrix}$ : $\mathbf{q}_1=\begin{pmatrix}1\\0\\0\end{pmatrix}$ (already a unit vector).

$\mathbf{v}_2=\begin{pmatrix}1\\1\\0\end{pmatrix}$ : $\langle\mathbf{v}_2,\mathbf{q}_1\rangle=1$ . $\mathbf{u}_2=\begin{pmatrix}1\\1\\0\end{pmatrix}-1\cdot\begin{pmatrix}1\\0\\0\end{pmatrix}=\begin{pmatrix}0\\1\\0\end{pmatrix}$ . $\mathbf{q}_2=\begin{pmatrix}0\\1\\0\end{pmatrix}$ .

$\mathbf{v}_3=\begin{pmatrix}1\\1\\1\end{pmatrix}$ : $\langle\mathbf{v}_3,\mathbf{q}_1\rangle=1$ , $\langle\mathbf{v}_3,\mathbf{q}_2\rangle=1$ .

$\mathbf{u}_3=\begin{pmatrix}1\\1\\1\end{pmatrix}-\begin{pmatrix}1\\0\\0\end{pmatrix}-\begin{pmatrix}0\\1\\0\end{pmatrix}=\begin{pmatrix}0\\0\\1\end{pmatrix}$ . $\mathbf{q}_3=\begin{pmatrix}0\\0\\1\end{pmatrix}$ .

$Q=I_3$ — the input vectors are already in the direction of the standard basis.

$R=\begin{pmatrix}1&1&1\\0&1&1\\0&0&1\end{pmatrix}$ (projection coefficients: $r_{11}=r_{22}=r_{33}=1$ , $r_{12}=1$ , $r_{13}=1$ , $r_{23}=1$ ).

Apply Gram-Schmidt to $\mathbf{v}_1=\begin{pmatrix}1\\0\\0\end{pmatrix}$ , $\mathbf{v}_2=\begin{pmatrix}1\\1\\0\end{pmatrix}$ , $\mathbf{v}_3=\begin{pmatrix}1\\1\\1\end{pmatrix}$ and write the resulting QR decomposition of the $3\times3$ matrix with these columns.

EXERCISE 11.4

Show that the orthogonal set produced by Gram-Schmidt is linearly independent using the property from Chapter 10: every orthogonal set of nonzero vectors is independent. Also show that $\text{span}\{\mathbf{q}_1,\ldots,\mathbf{q}_k\}=\text{span}\{\mathbf{v}_1,\ldots,\mathbf{v}_k\}$ by induction.

Independence of $\{\mathbf{q}_1,\ldots,\mathbf{q}_k\}$ : By construction, $\langle\mathbf{q}_i,\mathbf{q}_j\rangle=0$ for $i\neq j$ and $\|\mathbf{q}_i\|=1\neq0$ . Suppose $\sum_i c_i\mathbf{q}_i=\mathbf{0}$ . Take the inner product with $\mathbf{q}_j$ : $\sum_i c_i\langle\mathbf{q}_i,\mathbf{q}_j\rangle = c_j\cdot1 + \sum_{i\neq j}c_i\cdot0 = c_j = 0$ . So all $c_j=0$ — linearly independent.

Same span: Induction on $k$ . Base: $\text{span}\{\mathbf{q}_1\}=\text{span}\{\mathbf{v}_1\}$ since $\mathbf{q}_1=\mathbf{v}_1/\|\mathbf{v}_1\|$ . Inductive step: $\mathbf{q}_k = (\mathbf{v}_k - \sum_{i<k}\langle\mathbf{v}_k,\mathbf{q}_i\rangle\mathbf{q}_i)/\|\cdot\|$ is a linear combination of $\mathbf{v}_k$ and $\mathbf{q}_1,\ldots,\mathbf{q}_{k-1}$ , which by inductive hypothesis lie in $\text{span}\{\mathbf{v}_1,\ldots,\mathbf{v}_{k-1}\}$ . So $\mathbf{q}_k\in\text{span}\{\mathbf{v}_1,\ldots,\mathbf{v}_k\}$ . Conversely, $\mathbf{v}_k = \|\mathbf{u}_k\|\mathbf{q}_k+\sum_{i<k}\langle\mathbf{v}_k,\mathbf{q}_i\rangle\mathbf{q}_i \in \text{span}\{\mathbf{q}_1,\ldots,\mathbf{q}_k\}$ .

Prove that the Gram-Schmidt output $\{\mathbf{q}_1,\ldots,\mathbf{q}_k\}$ (a) is linearly independent and (b) spans the same space as $\{\mathbf{v}_1,\ldots,\mathbf{v}_k\}$ . Use properties of inner products and induction.

EXERCISE 11.5

In QR, the diagonal entries $r_{kk} = \|\mathbf{u}_k\|$ . A diagonal entry of $R$ is zero $\iff$ $\|\mathbf{u}_k\|=0$ $\iff$ $\mathbf{v}_k$ is in the span of $\mathbf{v}_1,\ldots,\mathbf{v}_{k-1}$ $\iff$ the columns of $A$ are linearly dependent.

If columns of $A$ are linearly independent: $\mathbf{v}_k \notin \text{span}\{\mathbf{v}_1,\ldots,\mathbf{v}_{k-1}\}$ for all $k$ , so $\mathbf{u}_k\neq\mathbf{0}$ for all $k$ , so $r_{kk}=\|\mathbf{u}_k\|>0$ — all diagonal entries are positive.

If some $r_{kk}=0$ : then $\|\mathbf{u}_k\|=0$ , meaning $\mathbf{v}_k = \sum_{i=1}^{k-1}\langle\mathbf{v}_k,\mathbf{q}_i\rangle\mathbf{q}_i$ — $\mathbf{v}_k$ is a linear combination of $\mathbf{v}_1,\ldots,\mathbf{v}_{k-1}$ , so the columns are linearly dependent.

Therefore: $A$ has linearly independent columns $\iff$ all diagonal entries of $R$ are nonzero $\iff$ $R$ is invertible. This gives a clean test for column independence from the QR factorisation.

Prove that the diagonal entries $r_{kk}$ of $R$ in the QR decomposition are all nonzero if and only if the columns of $A$ are linearly independent. What does a zero diagonal entry in $R$ reveal about the columns of $A$ ?

EXERCISE 11.6

Orthogonalise the factor columns using Gram-Schmidt. The first orthogonal factor is the market return; the second is the market-neutral component of the value factor. Then check that the resulting factors have zero correlation.

$\mathbf{f}_1=\begin{pmatrix}0.03\\0.01\\0.05\\0.02\end{pmatrix}$ (market), $\mathbf{f}_2=\begin{pmatrix}0.02\\0.03\\0.01\\0.04\end{pmatrix}$ (value).

$\|\mathbf{f}_1\|^2 = 0.0009+0.0001+0.0025+0.0004=0.0039$ .

$\langle\mathbf{f}_2,\mathbf{f}_1\rangle = 0.0006+0.0003+0.0005+0.0008=0.0022$ .

Projection coefficient: $\frac{0.0022}{0.0039}\approx0.564$ .

Orthogonalised value factor: $\mathbf{g}_2=\mathbf{f}_2-0.564\mathbf{f}_1=\begin{pmatrix}0.02-0.017\\0.03-0.006\\0.01-0.028\\0.04-0.011\end{pmatrix}\approx\begin{pmatrix}0.003\\0.024\\-0.018\\0.029\end{pmatrix}$ .

The original value factor $\mathbf{f}_2$ contains a market component (the projection). After removing it, $\mathbf{g}_2$ is the pure value signal — market-neutral. Its correlation with $\mathbf{f}_1$ is now exactly zero. Regression of returns on $(\mathbf{f}_1, \mathbf{g}_2)$ gives non-overlapping factor betas, so value and market contributions to returns are separately attributed.

A fund uses two return factors: market returns $\mathbf{f}_1=(0.03, 0.01, 0.05, 0.02)^T$ and value-factor returns $\mathbf{f}_2=(0.02, 0.03, 0.01, 0.04)^T$ over four periods. Apply one step of Gram-Schmidt to produce an orthogonalised value factor $\mathbf{g}_2$ that is uncorrelated with the market. Interpret what the orthogonalisation removes.

06 · Chapter Summary

Concept	Formula / Rule
Projection onto $\mathbf{u}$	$\text{proj}_\mathbf{u}\mathbf{v} = \frac{\langle\mathbf{v},\mathbf{u}\rangle}{\langle\mathbf{u},\mathbf{u}\rangle}\mathbf{u}$
Orthogonal remainder	$\mathbf{v}-\text{proj}_\mathbf{u}\mathbf{v}\perp\mathbf{u}$
Gram-Schmidt step $j$	$\mathbf{u}_j=\mathbf{v}_j-\sum_{i<j}\langle\mathbf{v}_j,\mathbf{q}_i\rangle\mathbf{q}_i$ ; $\mathbf{q}_j=\mathbf{u}_j/\\|\mathbf{u}_j\\|$
Fails when	Input is linearly dependent; $\mathbf{u}_j=\mathbf{0}$ for some $j$
QR decomposition	$A=QR$ ; $Q$ has orthonormal cols; $R$ upper triangular
$R$ diagonal entries	$r_{kk}=\\|\mathbf{u}_k\\|>0 \iff$ columns of $A$ are independent
Span preserved	$\text{span}\{\mathbf{q}_1,\ldots,\mathbf{q}_k\}=\text{span}\{\mathbf{v}_1,\ldots,\mathbf{v}_k\}$
Quant use	Orthogonal factor models; stable numerical least squares

Next: Chapter 12 — Projections & Least Squares derives the normal equations from the projection framework, showing that the ordinary least squares regression solution is the orthogonal projection of the data onto the column space of the design matrix.