Chapter 01

Vectors & Vector Spaces

00 · Symbol Glossary

Every symbol you'll see in this chapter, named and explained. When a new one appears, refer back here.

$\mathbb{R}$Blackboard R

The set of all real numbers — every number on the number line: negative, zero, positive, decimals, irrationals like $\pi$ . The double-struck style of $\mathbb{R}$ is standard in mathematics to distinguish it from an ordinary $R$ .

$\mathbb{R}^n$R-n

The set of all vectors with exactly $n$ real-number components. $\mathbb{R}^2$ is the 2D plane, $\mathbb{R}^3$ is 3D space, $\mathbb{R}^{100}$ is 100-dimensional space — used in machine learning constantly.

$\mathbf{v}$Bold v — vector

A vector. Bold lowercase letters always denote vectors. Plain lowercase like $v$ (no bold) typically denotes a scalar — a single number, not a list.

$v_i$v sub i — component

The $i$ -th entry of vector $\mathbf{v}$ . The subscript $i$ is an index that selects a slot. $v_1$ is the first entry, $v_2$ the second, $v_n$ the last.

$\mathbf{v}^T$v transpose

The superscript $T$ means transpose — flip the column vector into a row vector. A column $\begin{pmatrix}3\\5\end{pmatrix}$ becomes the row $\begin{pmatrix}3 & 5\end{pmatrix}$ . Critical for writing dot products as matrix multiplication.

$\in$"in" / element of

Membership. $\mathbf{v} \in \mathbb{R}^n$ reads "v is an element of $\mathbb{R}^n$ " — meaning v belongs to that space. Think of it as the mathematical word "in."

$\sum$Sigma — summation

Greek capital letter sigma. Means "add up a sequence." $\sum_{i=1}^{n} v_i$ means: start at $i=1$ , go up to $i=n$ , and add $v_1 + v_2 + \cdots + v_n$ . A compact way to write long sums.

$\|\mathbf{v}\|$Norm of v — length

The Euclidean length (magnitude) of a vector — how long the arrow is. The double vertical bars are the norm notation. Always a non-negative number.

$\mathbf{u} \cdot \mathbf{v}$Dot product

An operation between two vectors that returns a single number (a scalar). Computed by multiplying corresponding components and summing. The dot $\cdot$ distinguishes this from scalar multiplication.

$\mathbf{0}$Zero vector

The vector where every component is 0. Bold to distinguish from the number zero. $\mathbf{0} \in \mathbb{R}^n$ is an $n$ -component vector of zeros — the "do nothing" element of a vector space.

$\exists$"there exists"

Existential quantifier. $\exists\, \mathbf{0}$ means "there exists a zero vector." Used in formal definitions to assert that something exists without naming it explicitly.

$\forall$"for all"

Universal quantifier. $\forall\, \mathbf{v} \in V$ means "for every vector v in V" — the statement that follows must hold without exception.

$c,\ d$Scalars

Plain (non-bold) letters representing single real numbers. Called "scalars" to contrast with vectors. They scale vectors — stretching, shrinking, or flipping them.

$\theta$Theta — angle

Greek lowercase letter theta. In this chapter it represents the angle between two vectors. You'll encounter it again in the dot product's geometric formula.

$\cos\theta$Cosine of theta

A trigonometric function that measures how aligned two directions are. $\cos(0°) = 1$ (same direction), $\cos(90°) = 0$ (perpendicular), $\cos(180°) = -1$ (opposite). Appears inside the dot product's geometric formula.

$\sqrt{\phantom{x}}$Square root

$\sqrt{x}$ is the number which, multiplied by itself, gives $x$ . Example: $\sqrt{25} = 5$ because $5 \times 5 = 25$ . Used in the norm formula to "undo" the squaring of components.

$\hat{\mathbf{v}}$v-hat — unit vector

The hat accent means the vector has been normalized — its length is exactly 1. Direction is preserved, magnitude is removed. Read aloud as "v hat."

$\dim(V)$Dimension of V

The number of vectors in any basis of vector space $V$ . It measures the "degrees of freedom" — how many independent directions exist in the space.

$\text{span}\{\cdot\}$Span

The set of all possible linear combinations of the vectors inside the braces. Represents every point reachable by mixing those vectors with any real-number coefficients.

01 · What is a Vector?

A vector is an ordered list of numbers. "Ordered" means position matters — the first slot is different from the second slot.

Think of it as coordinates. If you say "I'm 3 blocks east and 5 blocks north," you've described a position using two numbers in a specific order — that's a vector: $\begin{pmatrix}3\\5\end{pmatrix}$ .

Definition — Vector in $\mathbb{R}^n$

An $n$ -dimensional vector is an ordered list of $n$ real numbers, written vertically as a column:

\mathbf{v} = \begin{pmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{pmatrix} \in \mathbb{R}^n

$\mathbf{v}$ — bold lowercase, the vector itself.

$v_1, v_2, \ldots, v_n$ — the individual numbers inside, called components or entries.

$\in \mathbb{R}^n$ — reads "is an element of R-n," meaning this vector lives in $n$ -dimensional real space.

$\vdots$ — vertical dots meaning "and so on, continuing the pattern."

Concrete Examples

Example — 2D Vector

A stock has return 4% and volatility 2%. As a vector in $\mathbb{R}^2$ :

\mathbf{v} = \begin{pmatrix} 4 \\ 2 \end{pmatrix}

First slot ( $v_1$ ) = 4 → return. Second slot ( $v_2$ ) = 2 → volatility. Order is fixed — swapping them would mean something completely different.

Example — 3D Vector

A point in 3D space at $x=1$ , $y=-3$ , $z=7$ :

\mathbf{p} = \begin{pmatrix} 1 \\ -3 \\ 7 \end{pmatrix} \in \mathbb{R}^3

$v_1 = 1$ , $v_2 = -3$ , $v_3 = 7$ . Negative numbers are perfectly valid.

Geometric Picture

In 2D, a vector $\begin{pmatrix}3\\5\end{pmatrix}$ is an arrow starting at the origin $(0,0)$ , pointing to the point $(3, 5)$ . The two numbers tell you how far right and how far up to go. This arrow interpretation is critical — it lets you visualize addition and scaling.

Row vs Column

Vectors written horizontally like $(3,\ 5)$ are row vectors. Vectors written vertically (as above) are column vectors. Default in linear algebra: column. When you see $\mathbf{v}^T$ (v-transpose), it means flip the column into a row: $\mathbf{v}^T = \begin{pmatrix}3 & 5\end{pmatrix}$ . This distinction matters a lot for matrix multiplication later.

02 · Vector Operations

Vector Addition

Add two vectors by adding their components in the same position. First slot with first slot, second with second, and so on. Both vectors must have the same number of components — you cannot add a 2D vector to a 3D vector.

Definition — Vector Addition

\mathbf{u} + \mathbf{v} = \begin{pmatrix}u_1\\u_2\\\vdots\\u_n\end{pmatrix} + \begin{pmatrix}v_1\\v_2\\\vdots\\v_n\end{pmatrix} = \begin{pmatrix}u_1+v_1\\u_2+v_2\\\vdots\\u_n+v_n\end{pmatrix}

Step-by-step — Adding $\mathbf{u} = \begin{pmatrix}2\\4\\-1\end{pmatrix}$ and $\mathbf{v} = \begin{pmatrix}3\\-2\\5\end{pmatrix}$

Confirm same dimension: both have 3 components.

\mathbf{u}, \mathbf{v} \in \mathbb{R}^3

. We can add them.

Add slot 1:

u_1 + v_1 = 2 + 3 = 5

. This becomes the first entry of the result.

Add slot 2:

u_2 + v_2 = 4 + (-2) = 2

. Adding a negative is the same as subtracting.

Add slot 3:

u_3 + v_3 = (-1) + 5 = 4

Assemble the result:

\mathbf{u}+\mathbf{v} = \begin{pmatrix}5\\2\\4\end{pmatrix}

Geometric meaning: place the tail of $\mathbf{v}$ at the tip of $\mathbf{u}$ . The result is the arrow from the origin to where $\mathbf{v}$ 's tip ends up. Like walking 2 blocks east then 3 blocks north — the total displacement is $2+3$ in each direction.

Common mistake — Mismatched Dimensions

You cannot add $\begin{pmatrix}2\\4\end{pmatrix} \in \mathbb{R}^2$ and $\begin{pmatrix}1\\3\\5\end{pmatrix} \in \mathbb{R}^3$ . There's no third slot in the first vector to pair with the $5$ . This is undefined — full stop.

Scalar Multiplication

Multiply a vector by a single number (a scalar). Multiply every component by that number. This stretches or shrinks — and possibly flips — the arrow.

Definition — Scalar Multiplication

c\mathbf{v} = c\begin{pmatrix}v_1\\v_2\\\vdots\\v_n\end{pmatrix} = \begin{pmatrix}cv_1\\cv_2\\\vdots\\cv_n\end{pmatrix}

$c$ is the scalar — just a real number. It's not bold because it's not a vector.

Step-by-step — Computing $-2 \cdot \mathbf{v}$ where $\mathbf{v} = \begin{pmatrix}3\\-1\\4\end{pmatrix}$

Identify scalar and vector: scalar

c = -2

, vector

\mathbf{v} = \begin{pmatrix}3\\-1\\4\end{pmatrix}

Multiply slot 1:

(-2) \times 3 = -6

. Negative times positive = negative.

Multiply slot 2:

(-2) \times (-1) = 2

. Negative times negative = positive.

Multiply slot 3:

(-2) \times 4 = -8

Result:

-2\mathbf{v} = \begin{pmatrix}-6\\2\\-8\end{pmatrix}

. The scalar

c = -2

doubled the length (

|-2| = 2

) and flipped direction (negative sign).

Special cases: $c = 0$ gives the zero vector $\mathbf{0}$ . $c = 1$ gives back $\mathbf{v}$ unchanged. $c = -1$ flips the arrow to point the opposite direction with the same length.

Dot Product

The most important operation in this chapter. Multiply corresponding components together, then add all those products up. The result is a single number (a scalar), not a vector.

Definition — Dot Product

\mathbf{u} \cdot \mathbf{v} = u_1 v_1 + u_2 v_2 + \cdots + u_n v_n = \sum_{i=1}^{n} u_i v_i

$\sum_{i=1}^{n}$ — "sum from $i=1$ to $n$ ." The index $i$ steps through $1, 2, 3, \ldots, n$ , and for each value of $i$ you compute $u_i v_i$ , then add them all.

$u_i v_i$ — the $i$ -th component of $\mathbf{u}$ multiplied by the $i$ -th component of $\mathbf{v}$ .

Step-by-step — $\mathbf{u} \cdot \mathbf{v}$ where $\mathbf{u} = \begin{pmatrix}2\\-1\\3\end{pmatrix}$, $\mathbf{v} = \begin{pmatrix}4\\5\\-2\end{pmatrix}$

Pair slot 1:

u_1 \times v_1 = 2 \times 4 = 8

. (First component of each vector.)

Pair slot 2:

u_2 \times v_2 = (-1) \times 5 = -5

. (Second component of each vector.)

Pair slot 3:

u_3 \times v_3 = 3 \times (-2) = -6

. (Third component of each vector.)

Sum all products:

\mathbf{u}\cdot\mathbf{v} = 8 + (-5) + (-6) = 8 - 5 - 6 = -3

. Result is

-3

, a scalar. The dot product is negative here — we'll see what that means geometrically next.

Geometric Meaning of the Dot Product

There is a second formula for the dot product that reveals its geometric meaning:

\mathbf{u} \cdot \mathbf{v} = \|\mathbf{u}\|\,\|\mathbf{v}\|\cos\theta

where $\theta$ is the angle between the two vectors, $\|\mathbf{u}\|$ and $\|\mathbf{v}\|$ are the lengths of the vectors (always positive), and $\cos\theta$ is the cosine of the angle.

Definition — What the sign of the dot product tells you

Positive: $\mathbf{u}\cdot\mathbf{v} > 0 \implies \cos\theta > 0 \implies \theta < 90°$ . Vectors point roughly in the same direction.

Zero: $\mathbf{u}\cdot\mathbf{v} = 0 \implies \cos\theta = 0 \implies \theta = 90°$ . Vectors are perpendicular (called orthogonal). This is hugely important in PCA.

Negative: $\mathbf{u}\cdot\mathbf{v} < 0 \implies \cos\theta < 0 \implies \theta > 90°$ . Vectors point in opposing directions.

Norm (Vector Length)

The norm is the Euclidean length of a vector — how long the arrow is. It's always a non-negative number.

Definition — Euclidean Norm (L2 Norm)

\|\mathbf{v}\| = \sqrt{v_1^2 + v_2^2 + \cdots + v_n^2} = \sqrt{\sum_{i=1}^{n} v_i^2}

$\|\mathbf{v}\|$ — double bars denote the norm. The "L2" label means we're squaring, summing, then square-rooting (as opposed to other norms which use different powers). This extends Pythagoras' theorem to $n$ dimensions.

Step-by-step — Norm of $\mathbf{v} = \begin{pmatrix}2\\-3\\6\end{pmatrix}$

Square each component:

v_1^2 = 2^2 = 4

v_2^2 = (-3)^2 = 9

v_3^2 = 6^2 = 36

. Note: squaring makes negatives positive, so

(-3)^2 = 9

not

-9

Sum the squares:

4 + 9 + 36 = 49

. This number came from:

4

(from squaring

2

)

+ 9

(from squaring

-3

)

+ 36

(from squaring

6

Take the square root:

\|\mathbf{v}\| = \sqrt{4 + 9 + 36} = \sqrt{49} = 7

, because

7 \times 7 = 49

Unit Vectors — Normalizing

A unit vector has norm exactly equal to 1. To convert any vector to a unit vector, divide by its norm. This preserves direction but removes magnitude.

Step-by-step — Normalize $\mathbf{v} = \begin{pmatrix}3\\4\end{pmatrix}$

Compute the norm:

\|\mathbf{v}\| = \sqrt{3^2 + 4^2} = \sqrt{9+16} = \sqrt{25} = 5

. Numbers

9

and

16

came from squaring

3

and

4

respectively.

Divide every component by 5:

\hat{\mathbf{v}} = \frac{\mathbf{v}}{\|\mathbf{v}\|} = \frac{1}{5}\begin{pmatrix}3\\4\end{pmatrix} = \begin{pmatrix}3/5\\4/5\end{pmatrix} = \begin{pmatrix}0.6\\0.8\end{pmatrix}

Verify:

\|\hat{\mathbf{v}}\| = \sqrt{(0.6)^2 + (0.8)^2} = \sqrt{0.36 + 0.64} = \sqrt{1} = 1

03 · Linear Combinations

A linear combination means: take a set of vectors, multiply each by a scalar, and add the results. You're mixing vectors together. This is the single most fundamental operation in linear algebra — everything else builds on it.

Definition — Linear Combination

c_1\mathbf{v}_1 + c_2\mathbf{v}_2 + \cdots + c_k\mathbf{v}_k = \sum_{i=1}^{k} c_i\mathbf{v}_i

$c_1, c_2, \ldots, c_k$ — scalars called coefficients. They control how much of each vector you use.

$\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_k$ — the vectors being combined. The subscripts here are labels (vector 1, vector 2...), not component indices.

$k$ — the number of vectors in the combination.

Step-by-step — $3\mathbf{v}_1 - 2\mathbf{v}_2$ where $\mathbf{v}_1=\begin{pmatrix}1\\0\end{pmatrix}$, $\mathbf{v}_2=\begin{pmatrix}2\\1\end{pmatrix}$

Identify coefficients and vectors:

c_1 = 3

with

\mathbf{v}_1

, and

c_2 = -2

with

\mathbf{v}_2

Scale $\mathbf{v}_1$ by 3:

3\mathbf{v}_1 = 3\begin{pmatrix}1\\0\end{pmatrix} = \begin{pmatrix}3\\0\end{pmatrix}

Scale $\mathbf{v}_2$ by $-2$ :

-2\mathbf{v}_2 = -2\begin{pmatrix}2\\1\end{pmatrix} = \begin{pmatrix}-4\\-2\end{pmatrix}

Add the scaled vectors component-by-component:

\begin{pmatrix}3\\0\end{pmatrix} + \begin{pmatrix}-4\\-2\end{pmatrix} = \begin{pmatrix}3+(-4)\\0+(-2)\end{pmatrix} = \begin{pmatrix}-1\\-2\end{pmatrix}

This result $\begin{pmatrix}-1\\-2\end{pmatrix}$ is one particular linear combination. By changing $c_1$ and $c_2$ to any real numbers, you can produce infinitely many different vectors — this is the idea behind span.

04 · Span

Definition — Span

The span of vectors $\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}$ is the set of all vectors you can produce by forming every possible linear combination:

\text{span}\{\mathbf{v}_1, \ldots, \mathbf{v}_k\} = \left\{\, c_1\mathbf{v}_1 + \cdots + c_k\mathbf{v}_k \;\middle|\; c_1, \ldots, c_k \in \mathbb{R} \,\right\}

The set-builder notation reads: "the set of all things of the form [left side] where [right side] is the condition." Here: all linear combinations where the scalars are real numbers.

Example — Single Vector Spans a Line

What is $\text{span}\left\{\begin{pmatrix}1\\2\end{pmatrix}\right\}$ ?

Only one vector, so any linear combination is just $c \cdot \begin{pmatrix}1\\2\end{pmatrix} = \begin{pmatrix}c\\2c\end{pmatrix}$ for any $c \in \mathbb{R}$ .

This traces out a line through the origin in the direction $(1, 2)$ . Points like $\begin{pmatrix}2\\4\end{pmatrix}$ , $\begin{pmatrix}-3\\-6\end{pmatrix}$ , $\begin{pmatrix}0\\0\end{pmatrix}$ are all on this line — all in the span. But $\begin{pmatrix}1\\3\end{pmatrix}$ is NOT, because $c = 1$ gives $\begin{pmatrix}1\\2\end{pmatrix}$ , not $\begin{pmatrix}1\\3\end{pmatrix}$ .

Example — Two Non-Parallel Vectors Span a Plane

What is $\text{span}\left\{\begin{pmatrix}1\\0\end{pmatrix}, \begin{pmatrix}0\\1\end{pmatrix}\right\}$ ?

Any combination: $c_1\begin{pmatrix}1\\0\end{pmatrix} + c_2\begin{pmatrix}0\\1\end{pmatrix} = \begin{pmatrix}c_1\\c_2\end{pmatrix}$ . Since $c_1$ and $c_2$ are any real numbers, this produces every point in $\mathbb{R}^2$ . The span is the entire 2D plane.

Common mistake — Parallel Vectors Have Limited Span

$\text{span}\left\{\begin{pmatrix}1\\2\end{pmatrix}, \begin{pmatrix}2\\4\end{pmatrix}\right\}$ — notice that $\begin{pmatrix}2\\4\end{pmatrix} = 2\begin{pmatrix}1\\2\end{pmatrix}$ . These are parallel (same direction, different length). Any combination $c_1\begin{pmatrix}1\\2\end{pmatrix} + c_2\begin{pmatrix}2\\4\end{pmatrix} = (c_1 + 2c_2)\begin{pmatrix}1\\2\end{pmatrix}$ is still just a multiple of the first vector. Span is still only a line, not the plane — the second vector adds no new direction.

05 · Vector Spaces

A vector space is a set $V$ with two operations (addition and scalar multiplication) that satisfy 8 rules. The key idea: these operations must never take you outside the set — the space is self-contained.

Definition — Vector Space

A set $V$ over $\mathbb{R}$ is a vector space if $\forall\, \mathbf{u}, \mathbf{v} \in V$ and $\forall\, c \in \mathbb{R}$ :

\mathbf{u} + \mathbf{v} \in V \qquad \text{(closed under addition)}

$\forall$ — "for all." This must hold for every possible pair of vectors, not just some.

Closed — the result stays inside $V$ . The space doesn't "leak."

The 8 Axioms

These aren't arbitrary rules. Each one captures something that must be true for the math to be consistent and useful.

Axiom	Rule	Why it matters
1. Commutativity	$\mathbf{u}+\mathbf{v} = \mathbf{v}+\mathbf{u}$	Order of addition doesn't matter — walking east then north = north then east.
2. Associativity	$(\mathbf{u}+\mathbf{v})+\mathbf{w} = \mathbf{u}+(\mathbf{v}+\mathbf{w})$	Grouping doesn't matter; lets you drop parentheses.
3. Zero vector	$\exists\,\mathbf{0} \in V$ such that $\mathbf{v}+\mathbf{0}=\mathbf{v}$	A "do nothing" element exists — the origin.
4. Additive inverse	$\forall\,\mathbf{v},\, \exists\,(-\mathbf{v})$ such that $\mathbf{v}+(-\mathbf{v})=\mathbf{0}$	Every vector has an opposite; allows subtraction.
5. Scalar identity	$1\cdot\mathbf{v} = \mathbf{v}$	Multiplying by 1 is neutral.
6. Scalar associativity	$(cd)\mathbf{v} = c(d\mathbf{v})$	Scaling twice = scaling by product; $2 \times 3 = 6$ .
7. Distributive (vector)	$c(\mathbf{u}+\mathbf{v}) = c\mathbf{u}+c\mathbf{v}$	Scalar distributes over vector sum.
8. Distributive (scalar)	$(c+d)\mathbf{v} = c\mathbf{v}+d\mathbf{v}$	Vector distributes over scalar sum.

Example — $\mathbb{R}^2$ Is a Vector Space

Take $\mathbf{u} = \begin{pmatrix}1\\3\end{pmatrix}$ and $\mathbf{v} = \begin{pmatrix}-2\\4\end{pmatrix}$ , both in $\mathbb{R}^2$ .

$\mathbf{u}+\mathbf{v} = \begin{pmatrix}-1\\7\end{pmatrix} \in \mathbb{R}^2$ — still a 2D vector.

$5\mathbf{v} = \begin{pmatrix}-10\\20\end{pmatrix} \in \mathbb{R}^2$ — still a 2D vector.

All 8 axioms hold. $\mathbb{R}^2$ is a vector space.

Common mistake — The First Quadrant Is Not a Vector Space

Let $V$ be all vectors in $\mathbb{R}^2$ with non-negative entries: $x \geq 0$ , $y \geq 0$ — the first quadrant.

Take $\mathbf{v} = \begin{pmatrix}1\\2\end{pmatrix} \in V$ . Compute $(-1)\mathbf{v} = \begin{pmatrix}-1\\-2\end{pmatrix}$ . Both components are negative, so $\begin{pmatrix}-1\\-2\end{pmatrix}$ is not in $V$ .

Axiom 4 (additive inverse) fails — $-\mathbf{v}$ is not in the set. $V$ is not a vector space.

06 · Subspaces

A subspace is a subset $W \subseteq V$ that is itself a vector space under the same operations. Instead of checking all 8 axioms (they're inherited from $V$ ), you only need 3 checks.

Definition — Subspace Test (3 Conditions)

$W$ is a subspace of $V$ if and only if:

S1. $\mathbf{0} \in W$ — the zero vector is in $W$ .

S2. $\mathbf{u}, \mathbf{v} \in W \implies \mathbf{u}+\mathbf{v} \in W$ — closed under addition.

S3. $\mathbf{v} \in W,\, c \in \mathbb{R} \implies c\mathbf{v} \in W$ — closed under scalar multiplication.

Step-by-step — Is the x-axis a subspace of $\mathbb{R}^2$?

Set up: The x-axis in

\mathbb{R}^2

is all vectors of the form

\begin{pmatrix}x\\0\end{pmatrix}

where

x \in \mathbb{R}

. Call this set

W

Check S1: Does

W

contain

\mathbf{0}

? Set

x=0

\begin{pmatrix}0\\0\end{pmatrix} \in W

. Yes.

Check S2: Take any two elements:

\begin{pmatrix}a\\0\end{pmatrix}+\begin{pmatrix}b\\0\end{pmatrix} = \begin{pmatrix}a+b\\0\end{pmatrix}

. Second component is still

0

, so the result is in

W

. Yes.

Check S3:

c\begin{pmatrix}a\\0\end{pmatrix} = \begin{pmatrix}ca\\0\end{pmatrix}

. Second component stays

0

. Still in

W

. Yes.

Conclusion: All three conditions pass. The x-axis is a subspace of

\mathbb{R}^2

Common mistake — A Line Not Through the Origin Is Not a Subspace

Let $W = \left\{\begin{pmatrix}x\\x+1\end{pmatrix} \mid x \in \mathbb{R}\right\}$ — a line shifted up by 1.

Check S1: Does $\begin{pmatrix}0\\0\end{pmatrix} \in W$ ? We'd need $x$ such that $x=0$ and $x+1=0$ simultaneously. Impossible — $0+1=1 \neq 0$ . Zero vector is NOT in $W$ .

Conclusion: Fails the very first test. Not a subspace. Every subspace must contain the origin.

07 · Linear Independence

Vectors are linearly independent if none of them can be built from linear combinations of the others. Each vector contributes a genuinely new direction that the others can't replicate.

Intuition: if you're describing a route, you don't need "go east" if you already have "go north" and "go northeast" — northeast is a combination of the other two. That redundancy = linear dependence.

Definition — Linear Independence

Vectors $\{\mathbf{v}_1, \ldots, \mathbf{v}_k\}$ are linearly independent if the only solution to:

c_1\mathbf{v}_1 + c_2\mathbf{v}_2 + \cdots + c_k\mathbf{v}_k = \mathbf{0}

is $c_1 = c_2 = \cdots = c_k = 0$ (all coefficients must be zero).

If any non-zero solution exists, the vectors are linearly dependent.

Why this definition? If one vector, say $\mathbf{v}_3$ , equals $2\mathbf{v}_1 - \mathbf{v}_2$ , then $2\mathbf{v}_1 - \mathbf{v}_2 - \mathbf{v}_3 = \mathbf{0}$ — a non-zero solution with $c_1=2, c_2=-1, c_3=-1$ . The equation catches the redundancy.

Example — Linearly Independent Vectors

Test $\mathbf{v}_1 = \begin{pmatrix}1\\0\end{pmatrix}$ , $\mathbf{v}_2 = \begin{pmatrix}0\\1\end{pmatrix}$ .

Set up: $c_1\begin{pmatrix}1\\0\end{pmatrix} + c_2\begin{pmatrix}0\\1\end{pmatrix} = \begin{pmatrix}0\\0\end{pmatrix}$ .

Computing the left side: $\begin{pmatrix}c_1\\0\end{pmatrix} + \begin{pmatrix}0\\c_2\end{pmatrix} = \begin{pmatrix}c_1\\c_2\end{pmatrix} = \begin{pmatrix}0\\0\end{pmatrix}$ .

Match components: slot 1 gives $c_1 = 0$ . Slot 2 gives $c_2 = 0$ . The only solution is $c_1=c_2=0$ . Linearly independent.

Common mistake — Parallel Vectors Are Linearly Dependent

Test $\mathbf{v}_1=\begin{pmatrix}1\\2\end{pmatrix}$ , $\mathbf{v}_2=\begin{pmatrix}3\\6\end{pmatrix}$ .

Notice: $\mathbf{v}_2 = 3\mathbf{v}_1$ — every component of $\mathbf{v}_2$ is exactly 3 times $\mathbf{v}_1$ . They're parallel.

Try $c_1 = 3$ , $c_2 = -1$ : $3\begin{pmatrix}1\\2\end{pmatrix} + (-1)\begin{pmatrix}3\\6\end{pmatrix} = \begin{pmatrix}3\\6\end{pmatrix} - \begin{pmatrix}3\\6\end{pmatrix} = \begin{pmatrix}0\\0\end{pmatrix}$ .

A non-zero solution exists ( $c_1=3, c_2=-1$ ). Linearly dependent. $\mathbf{v}_2$ is redundant — it adds no new direction.

08 · Basis & Dimension

A basis is the most efficient set of vectors that fully describes a vector space — no redundancy, no gaps. Think of it as the minimal vocabulary needed to express every vector in the space.

Definition — Basis

A set $B = \{\mathbf{b}_1, \ldots, \mathbf{b}_k\}$ is a basis for vector space $V$ if:

B1. $B$ is linearly independent (no redundancy).

B2. $\text{span}(B) = V$ (every vector in $V$ can be reached).

Example — Standard Basis of $\mathbb{R}^2$

The standard basis is:

\mathbf{e}_1 = \begin{pmatrix}1\\0\end{pmatrix}, \qquad \mathbf{e}_2 = \begin{pmatrix}0\\1\end{pmatrix}

Independent? Yes — shown above. Spans $\mathbb{R}^2$ ? Yes — any vector $\begin{pmatrix}x\\y\end{pmatrix} = x\mathbf{e}_1 + y\mathbf{e}_2$ , so $c_1=x$ , $c_2=y$ gives every point. It's a basis.

The letters $\mathbf{e}$ stand for "elementary" or "standard." The subscript tells which slot is $1$ — all other slots are $0$ .

Bases are not unique

$\left\{\begin{pmatrix}1\\1\end{pmatrix}, \begin{pmatrix}1\\-1\end{pmatrix}\right\}$ is also a valid basis for $\mathbb{R}^2$ — linearly independent and spans the plane. A space has infinitely many bases, but all bases for the same space have the same number of vectors. That number is the dimension.

Definition — Dimension

The dimension of a vector space $V$ , written $\dim(V)$ , is the number of vectors in any basis of $V$ . It is always the same regardless of which basis you choose.

\dim(\mathbb{R}^n) = n

A line through the origin has $\dim = 1$ . A plane has $\dim = 2$ . In machine learning, a dataset with 500 features lives in $\mathbb{R}^{500}$ , $\dim = 500$ — until PCA reduces it.

09 · Exercises

EXERCISE 1.1

Scale $\mathbf{v}$ by 3 first (multiply every component by 3), then add component-by-component to $\mathbf{u}$ .

Compute $3\mathbf{v} = 3\begin{pmatrix}1\\2\\-3\end{pmatrix} = \begin{pmatrix}3\\6\\-9\end{pmatrix}$ .

Then add component-by-component: $\mathbf{u}+3\mathbf{v} = \begin{pmatrix}2+3\\-1+6\\4+(-9)\end{pmatrix} = \begin{pmatrix}5\\5\\-5\end{pmatrix}$ .

Compute $\mathbf{u} + 3\mathbf{v}$ where $\mathbf{u}=\begin{pmatrix}2\\-1\\4\end{pmatrix}$ and $\mathbf{v}=\begin{pmatrix}1\\2\\-3\end{pmatrix}$ . Show every component step.

EXERCISE 1.2

Multiply corresponding components and sum. The sign of the result tells you the angle type: positive means acute, zero means right angle, negative means obtuse.

Compute each product and sum: $2(1) + (-1)(2) + 4(-3) = 2 - 2 - 12 = -12$ .

The dot product is $-12$ . Since it is negative, $\cos\theta < 0$ , which means $\theta > 90°$ — the angle is obtuse.

Find the dot product $\mathbf{u}\cdot\mathbf{v}$ for $\mathbf{u}=\begin{pmatrix}2\\-1\\4\end{pmatrix}$ and $\mathbf{v}=\begin{pmatrix}1\\2\\-3\end{pmatrix}$ . Is the angle between them acute, right, or obtuse?

EXERCISE 1.3

Square each component, sum the squares, then take the square root to get the norm. Then divide every component by the norm to obtain the unit vector.

Square each component: $0^2 + (-5)^2 + 12^2 = 0 + 25 + 144 = 169$ .

Take the square root: $\|\mathbf{w}\| = \sqrt{169} = 13$ .

Divide every component by 13: $\hat{\mathbf{w}} = \frac{1}{13}\begin{pmatrix}0\\-5\\12\end{pmatrix} = \begin{pmatrix}0\\-5/13\\12/13\end{pmatrix}$ .

Compute $\|\mathbf{w}\|$ for $\mathbf{w}=\begin{pmatrix}0\\-5\\12\end{pmatrix}$ , then find the unit vector $\hat{\mathbf{w}}$ .

EXERCISE 1.4

You need a scalar $c$ such that $c\begin{pmatrix}2\\-1\end{pmatrix}=\begin{pmatrix}4\\-2\end{pmatrix}$ . Try solving for $c$ using the first component, then verify the second component is also satisfied.

From the first component: $2c = 4$ , so $c = 2$ .

Verify with the second component: $(-1)(2) = -2$ . That matches the second component of $\begin{pmatrix}4\\-2\end{pmatrix}$ .

Yes, $\begin{pmatrix}4\\-2\end{pmatrix}$ is in $\text{span}\left\{\begin{pmatrix}2\\-1\end{pmatrix}\right\}$ with $c = 2$ .

Is $\begin{pmatrix}4\\-2\end{pmatrix}$ in $\text{span}\left\{\begin{pmatrix}2\\-1\end{pmatrix}\right\}$ ? Show why.

EXERCISE 1.5

Set up $c_1\mathbf{v}_1 + c_2\mathbf{v}_2 + c_3\mathbf{v}_3 = \mathbf{0}$ and check if a non-zero solution exists. Hint: notice whether $\mathbf{v}_3 = \mathbf{v}_1 + \mathbf{v}_2$ .

Observe that $\mathbf{v}_3 = \begin{pmatrix}1\\1\\2\end{pmatrix} = \begin{pmatrix}1\\0\\1\end{pmatrix} + \begin{pmatrix}0\\1\\1\end{pmatrix} = \mathbf{v}_1 + \mathbf{v}_2$ .

Therefore $1 \cdot \mathbf{v}_1 + 1 \cdot \mathbf{v}_2 + (-1) \cdot \mathbf{v}_3 = \mathbf{0}$ is a non-zero solution ( $c_1=1, c_2=1, c_3=-1$ ).

Linearly dependent. $\mathbf{v}_3$ is redundant — it is exactly the sum of the other two vectors.

Are $\mathbf{v}_1=\begin{pmatrix}1\\0\\1\end{pmatrix}$ , $\mathbf{v}_2=\begin{pmatrix}0\\1\\1\end{pmatrix}$ , $\mathbf{v}_3=\begin{pmatrix}1\\1\\2\end{pmatrix}$ linearly independent?

EXERCISE 1.6

The dot product $\mathbf{w}\cdot\mathbf{r} = \sum w_i r_i$ is a weighted average of the returns. To verify the weights are valid, check that $\mathbf{w}\cdot\mathbf{1} = 0.4 + 0.3 + 0.3 = 1$ .

Compute each product and sum: $0.4(0.08) + 0.3(0.05) + 0.3(0.12) = 0.032 + 0.015 + 0.036 = 0.083$ .

Portfolio expected return = 8.3%.

Weights check: $0.4 + 0.3 + 0.3 = 1.0$ — fully invested, no leverage.

A bond portfolio has positions in three assets with weights $\mathbf{w}=\begin{pmatrix}0.4\\0.3\\0.3\end{pmatrix}$ and expected returns $\mathbf{r}=\begin{pmatrix}0.08\\0.05\\0.12\end{pmatrix}$ . Compute the portfolio expected return $\mathbf{w}\cdot\mathbf{r}$ using the dot product, and verify that the weights form a valid portfolio ( $\mathbf{w}\cdot\mathbf{1}=1$ ).

10 · Chapter Summary

Concept	Formula / Rule
Vector in $\mathbb{R}^n$	Ordered list of $n$ reals; bold $\mathbf{v}$
Addition	Component-wise; same dimension required
Scalar mult.	Multiply every component by $c$
Dot product	$\mathbf{u}\cdot\mathbf{v}=\sum u_i v_i$ ; zero means orthogonal
Norm	$\\|\mathbf{v}\\|=\sqrt{\sum v_i^2}$ ; length of arrow
Unit vector	$\hat{\mathbf{v}}=\mathbf{v}/\\|\mathbf{v}\\|$ ; norm = 1
Linear combination	$\sum c_i\mathbf{v}_i$ ; the core building block
Span	All reachable by linear combinations
Vector space	Closed under + and scalar mult.; 8 axioms
Subspace test	Contains $\mathbf{0}$ ; closed under + and scalar mult.
Linear independence	Only zero solution to $\sum c_i\mathbf{v}_i=\mathbf{0}$
Basis	Independent set that spans the space
Dimension	Number of vectors in any basis

Next: Chapter 02 — Matrix Operations extends vectors into rectangular grids of numbers, defining addition, scalar multiplication, and the powerful (but non-commutative) matrix product.