Chapter 19

Markov Chains & Steady States

00 · Symbol Glossary

$P$P — transition matrix

A stochastic matrix whose rows sum to 1: each row gives the probability distribution of the next state given the current state. Entry $p_{ij}$ is the probability of moving from state $i$ to state $j$ in one step. Powers $P^n$ give $n$ -step transition probabilities.

$\boldsymbol{\pi}$pi — stationary distribution

A row vector $\boldsymbol{\pi}$ satisfying $\boldsymbol{\pi} P = \boldsymbol{\pi}$ and $\sum_i \pi_i = 1$ . Read aloud as "pi." The chain is in steady state when the state distribution equals $\boldsymbol{\pi}$ — the distribution does not change from one step to the next.

$X_t$X sub t — state at time t

The random state of the Markov chain at time $t$ . The Markov property: $P(X_{t+1} \mid X_t, X_{t-1}, \ldots) = P(X_{t+1} \mid X_t)$ — only the current state matters for predicting the future.

$P^n$P to the n — n-step transitions

The $n$ -th power of the transition matrix. Entry $(P^n)_{ij}$ is the probability of being in state $j$ after $n$ steps, starting from state $i$ . Computed via matrix multiplication — the same machinery from Chapter 2.

01 · Stochastic Matrices

A credit rating, a trading regime, or a customer's account status tomorrow depends only on today's state — not the full history. Markov chains encode this memoryless structure as matrix algebra.

Definition — Stochastic Matrix

A square matrix $P \in \mathbb{R}^{n \times n}$ is stochastic (or a transition matrix) if:

p_{ij} \geq 0 \text{ for all } i,j \qquad \sum_{j=1}^{n} p_{ij} = 1 \text{ for all } i

$p_{ij}$ — probability of transitioning from state $i$ to state $j$ in one step.

Each row is a probability distribution over the $n$ states. Entries are nonnegative; rows sum to exactly 1.

✓ Example — Credit Rating Transition Matrix

A simplified 3-state credit model: states $\{$ AAA, BBB, Default $\}$ .

P = \begin{pmatrix}0.92 & 0.07 & 0.01 \\ 0.02 & 0.90 & 0.08 \\ 0 & 0 & 1\end{pmatrix}

Row 1 (AAA): $92\%$ stay AAA, $7\%$ downgrade to BBB, $1\%$ default. Row 3 (Default): $100\%$ remain in default — an absorbing state. Each row sums to $0.92+0.07+0.01=1$ ✓.

❌ Failure — Rows Not Summing to 1

$P = \begin{pmatrix}0.5&0.3\\0.4&0.4\end{pmatrix}$ . Row 1 sums to $0.8$ ; row 2 sums to $0.8$ .

Why it breaks: $p_{ij}$ are supposed to be probabilities. If row $i$ sums to $0.8$ , there is a $20\%$ probability of transitioning to no state — undefined. Stochastic matrices require $\sum_j p_{ij} = 1$ for every row.

Consequence: $P^n$ no longer represents valid $n$ -step probabilities. Normalise each row before any Markov analysis.

02 · Markov Chains and the Markov Property

Definition — Homogeneous Markov Chain

A sequence of random variables $X_0, X_1, X_2, \ldots$ taking values in $\{1, \ldots, n\}$ is a Markov chain with transition matrix $P$ if:

P(X_{t+1} = j \mid X_t = i, X_{t-1}, \ldots, X_0) = p_{ij}

The future depends on the past only through the present state — the Markov property.

If the initial distribution is $\boldsymbol{\mu}^{(0)}$ (row vector), the distribution after $t$ steps is:

\boldsymbol{\mu}^{(t)} = \boldsymbol{\mu}^{(0)} P^t

Step-by-step — Two-step transitions for $P = \begin{pmatrix}0.5&0.5\\0.2&0.8\end{pmatrix}$, starting from state 1

One-step from state 1: $\boldsymbol{\mu}^{(1)} = \begin{pmatrix}1&0\end{pmatrix} P = \begin{pmatrix}0.5&0.5\end{pmatrix}$ .

The $0.5$ in position 1 comes from $1 \cdot p_{11} = 0.5$ ; the $0.5$ in position 2 from $1 \cdot p_{12} = 0.5$ .

Compute $P^2$ : $P^2 = PP$ .

$(P^2)_{11} = 0.5(0.5)+0.5(0.2) = 0.25+0.10 = 0.35$ .

$(P^2)_{12} = 0.5(0.5)+0.5(0.8) = 0.25+0.40 = 0.65$ .

$(P^2)_{21} = 0.2(0.5)+0.8(0.2) = 0.10+0.16 = 0.26$ .

$(P^2)_{22} = 0.2(0.5)+0.8(0.8) = 0.10+0.64 = 0.74$ .

Two-step from state 1: $\boldsymbol{\mu}^{(2)} = \begin{pmatrix}1&0\end{pmatrix} P^2 = \begin{pmatrix}0.35&0.65\end{pmatrix}$ .

Alternatively: $\boldsymbol{\mu}^{(2)} = \boldsymbol{\mu}^{(1)} P = \begin{pmatrix}0.5&0.5\end{pmatrix}\begin{pmatrix}0.5&0.5\\0.2&0.8\end{pmatrix} = \begin{pmatrix}0.35&0.65\end{pmatrix}$ ✓.

03 · Stationary Distribution

Definition — Stationary Distribution

A probability row vector $\boldsymbol{\pi} = (\pi_1, \ldots, \pi_n)$ is a stationary distribution of $P$ if:

\boldsymbol{\pi} P = \boldsymbol{\pi} \qquad \sum_{i=1}^{n} \pi_i = 1, \quad \pi_i \geq 0

$\boldsymbol{\pi}$ — fixed point of $P$ under left-multiplication. If $\boldsymbol{\mu}^{(0)} = \boldsymbol{\pi}$ , then $\boldsymbol{\mu}^{(t)} = \boldsymbol{\pi}$ for all $t$ .

Equivalently, $\boldsymbol{\pi}^T$ is an eigenvector of $P^T$ (or $P$ if $P$ is symmetric) with eigenvalue $\lambda = 1$ .

Step-by-step — Finding $\boldsymbol{\pi}$ for $P = \begin{pmatrix}0.5&0.5\\0.2&0.8\end{pmatrix}$

Set up $\boldsymbol{\pi} P = \boldsymbol{\pi}$ : let $\boldsymbol{\pi} = (\pi_1, \pi_2)$ .

$\pi_1' = 0.5\pi_1 + 0.2\pi_2 = \pi_1$ .

$\pi_2' = 0.5\pi_1 + 0.8\pi_2 = \pi_2$ .

Both equations must hold.

Solve the first equation: $0.5\pi_1 + 0.2\pi_2 = \pi_1 \Rightarrow 0.2\pi_2 = 0.5\pi_1 \Rightarrow \pi_2 = 2.5\pi_1$ .

The $2.5$ comes from $0.5/0.2 = 2.5$ .

Normalise: $\pi_1 + \pi_2 = 1$ . Substitute $\pi_2 = 2.5\pi_1$ : $\pi_1 + 2.5\pi_1 = 3.5\pi_1 = 1$ .

$\pi_1 = 1/3.5 = 2/7 \approx 0.286$ . $\pi_2 = 2.5/3.5 = 5/7 \approx 0.714$ .

Verify:

\boldsymbol{\pi} = \begin{pmatrix}\frac{2}{7} & \frac{5}{7}\end{pmatrix}

$\boldsymbol{\pi} P = \begin{pmatrix}\frac{2}{7}(0.5)+\frac{5}{7}(0.2) & \frac{2}{7}(0.5)+\frac{5}{7}(0.8)\end{pmatrix} = \begin{pmatrix}\frac{1}{7}+\frac{1}{7} & \frac{1}{7}+\frac{4}{7}\end{pmatrix} = \begin{pmatrix}\frac{2}{7} & \frac{5}{7}\end{pmatrix} = \boldsymbol{\pi}$ ✓.

❌ Failure — Confusing Stationary with Initial

For $P = \begin{pmatrix}0&1\\1&0\end{pmatrix}$ (deterministic alternation), $\boldsymbol{\pi} = (0.5, 0.5)$ is stationary: $\boldsymbol{\pi} P = (0.5, 0.5)$ ✓.

But starting from $\boldsymbol{\mu}^{(0)} = (1, 0)$ : $\boldsymbol{\mu}^{(1)} = (0,1)$ , $\boldsymbol{\mu}^{(2)} = (1,0)$ — the chain oscillates, never converging to $(0.5, 0.5)$ .

Consequence: $\boldsymbol{\pi} P = \boldsymbol{\pi}$ is necessary but not sufficient for convergence. A regular chain (some power $P^k$ has all positive entries) guarantees $\boldsymbol{\mu}^{(t)} \to \boldsymbol{\pi}$ from any start.

04 · Eigenvector Characterisation and $P^n$

Definition — Stationary Distribution as Eigenvector

$\boldsymbol{\pi} P = \boldsymbol{\pi}$ is equivalent to:

P^T \boldsymbol{\pi}^T = \boldsymbol{\pi}^T

$\boldsymbol{\pi}^T$ is an eigenvector of $P^T$ with eigenvalue 1. Finding $\boldsymbol{\pi}$ reduces to solving $(P^T - I)\boldsymbol{\pi}^T = \mathbf{0}$ with the normalisation $\sum \pi_i = 1$ — a linear algebra problem from Chapters 3 and 6.

Definition — Convergence to Steady State

If $P$ is regular ( $P^k$ entrywise positive for some $k$ ), then:

\lim_{n \to \infty} P^n = \mathbf{1}\,\boldsymbol{\pi}

$\mathbf{1}$ — column vector of ones. Every row of $P^n$ converges to $\boldsymbol{\pi}$ . The initial state is forgotten in the long run.

✓ Example — Regime-Switching Market Model

A 2-state market: Bull (B) and Bear (b). $P = \begin{pmatrix}0.9&0.1\\0.3&0.7\end{pmatrix}$ .

$\boldsymbol{\pi}$ : solve $0.9\pi_1+0.3\pi_2=\pi_1$ and $\pi_1+\pi_2=1$ . Get $\pi_1 = 0.75$ , $\pi_2 = 0.25$ .

Long-run: $75\%$ of time in Bull, $25\%$ in Bear — regardless of today's regime. Option pricing models (Hamilton regime-switching) use this structure to blend bull and bear volatility parameters.

05 · Absorbing States and Credit Risk

Definition — Absorbing State

State $i$ is absorbing if $p_{ii} = 1$ (equivalently, $p_{ij} = 0$ for all $j \neq i$ ). Once entered, the chain never leaves.

For the credit matrix above, Default is absorbing: $p_{33} = 1$ . The probability of eventual default from state $i$ is the $i$ -to-3 entry of the fundamental matrix $(I - Q)^{-1}$ applied to the default column, where $Q$ is the submatrix of transient states.

✓ Example — Default Probability from AAA

With $P = \begin{pmatrix}0.92&0.07&0.01\\0.02&0.90&0.08\\0&0&1\end{pmatrix}$ , transient states are $\{$ AAA, BBB $\}$ , $Q = \begin{pmatrix}0.92&0.07\\0.02&0.90\end{pmatrix}$ , default column $\mathbf{r} = \begin{pmatrix}0.01\\0.08\end{pmatrix}$ .

$(I-Q) = \begin{pmatrix}0.08&-0.07\\-0.02&0.10\end{pmatrix}$ . $(I-Q)^{-1}\mathbf{r}$ gives default probabilities from AAA and BBB — standard in credit portfolio risk (CreditMetrics, Basel IRB).

06 · Practice Exercises

EXERCISE 19.1

Check each row sums to 1 and all entries $\geq 0$ . If both hold, $P$ is stochastic.

$P = \begin{pmatrix}0.3&0.7\\0.6&0.4\end{pmatrix}$ .

Row 1: $0.3+0.7=1$ ✓. Row 2: $0.6+0.4=1$ ✓. All entries $\geq 0$ ✓.

$P$ is stochastic.

Verify that $P = \begin{pmatrix}0.3&0.7\\0.6&0.4\end{pmatrix}$ is a stochastic matrix. Check both row sums and nonnegativity.

EXERCISE 19.2

Compute $P^2$ by matrix multiplication. Entry $(P^2)_{ij}$ is the two-step transition probability from $i$ to $j$ .

$P = \begin{pmatrix}0.5&0.5\\0.2&0.8\end{pmatrix}$ .

$(P^2)_{11} = 0.5(0.5)+0.5(0.2) = 0.35$ . $(P^2)_{12} = 0.5(0.5)+0.5(0.8) = 0.65$ .

$(P^2)_{21} = 0.2(0.5)+0.8(0.2) = 0.26$ . $(P^2)_{22} = 0.2(0.5)+0.8(0.8) = 0.74$ .

P^2 = \begin{pmatrix}0.35&0.65\\0.26&0.74\end{pmatrix}

Starting from state 1: $\boldsymbol{\mu}^{(2)} = \begin{pmatrix}1&0\end{pmatrix}P^2 = \begin{pmatrix}0.35&0.65\end{pmatrix}$ .

For $P = \begin{pmatrix}0.5&0.5\\0.2&0.8\end{pmatrix}$ , compute $P^2$ and the state distribution after 2 steps starting from state 1 with certainty.

EXERCISE 19.3

Solve $\boldsymbol{\pi}P=\boldsymbol{\pi}$ with $\pi_1+\pi_2=1$ . Use one equation from $\boldsymbol{\pi}P=\boldsymbol{\pi}$ plus the normalisation.

$0.3\pi_1+0.6\pi_2=\pi_1 \Rightarrow 0.6\pi_2=0.7\pi_1 \Rightarrow \pi_2=\frac{7}{6}\pi_1$ .

$\pi_1+\frac{7}{6}\pi_1=1 \Rightarrow \frac{13}{6}\pi_1=1 \Rightarrow \pi_1=\frac{6}{13}$ .

$\pi_2=\frac{7}{13}$ .

$\boldsymbol{\pi}=\begin{pmatrix}6/13&7/13\end{pmatrix}$ . Verify: $0.3\cdot\frac{6}{13}+0.6\cdot\frac{7}{13}=\frac{1.8+4.2}{13}=\frac{6}{13}$ ✓.

Find the stationary distribution $\boldsymbol{\pi}$ for $P = \begin{pmatrix}0.3&0.7\\0.6&0.4\end{pmatrix}$ . Show the normalisation step and verify $\boldsymbol{\pi}P=\boldsymbol{\pi}$ .

EXERCISE 19.4

Stationary $\boldsymbol{\pi}$ satisfies $P^T\boldsymbol{\pi}^T=\boldsymbol{\pi}^T$ — eigenvalue 1. Solve $(P^T-I)\mathbf{x}=\mathbf{0}$ and normalise.

$P^T = \begin{pmatrix}0.9&0.3\\0.1&0.7\end{pmatrix}$ . $(P^T-I) = \begin{pmatrix}-0.1&0.3\\0.1&-0.3\end{pmatrix}$ .

Row 1: $-0.1x_1+0.3x_2=0 \Rightarrow x_1=3x_2$ .

Normalise: $3\pi_2+\pi_2=1 \Rightarrow \pi_2=0.25$ , $\pi_1=0.75$ .

$\boldsymbol{\pi}=(0.75, 0.25)$ . Eigenvalue check: $P^T\begin{pmatrix}0.75\\0.25\end{pmatrix}=\begin{pmatrix}0.675+0.075\\0.075+0.175\end{pmatrix}=\begin{pmatrix}0.75\\0.25\end{pmatrix}$ ✓.

For $P = \begin{pmatrix}0.9&0.1\\0.3&0.7\end{pmatrix}$ , find $\boldsymbol{\pi}$ by solving the eigenvector equation $P^T\boldsymbol{\pi}^T=\boldsymbol{\pi}^T$ . Interpret $\pi_1$ as the long-run fraction of time in state 1.

EXERCISE 19.5

State 3 is absorbing ( $p_{33}=1$ ). No stationary distribution exists over all three states with positive mass on the absorbing state — mass flows into default and stays.

$P = \begin{pmatrix}0.92&0.07&0.01\\0.02&0.90&0.08\\0&0&1\end{pmatrix}$ .

For $\boldsymbol{\pi}P=\boldsymbol{\pi}$ : $\pi_3 = \pi_1(0)+\pi_2(0)+\pi_3(1) = \pi_3$ — always satisfied.

But $\pi_1,\pi_2$ must also satisfy $0.92\pi_1+0.02\pi_2=\pi_1$ and $0.07\pi_1+0.90\pi_2=\pi_2$ .

From first: $0.02\pi_2=0.08\pi_1$ , so $\pi_2=4\pi_1$ . Second: $0.07\pi_1+0.90(4\pi_1)=\pi_2=4\pi_1$ gives $3.67\pi_1=4\pi_1$ — contradiction unless $\pi_1=\pi_2=0$ .

The only stationary distribution with support on transient states is $\boldsymbol{\pi}=(0,0,1)$ — all mass in default. The chain eventually absorbs: $\lim_{n\to\infty} P^n$ has third column $(1,1,1)^T$ .

For the 3-state credit matrix $P = \begin{pmatrix}0.92&0.07&0.01\\0.02&0.90&0.08\\0&0&1\end{pmatrix}$ : identify the absorbing state. Explain why no stationary distribution with $\pi_1,\pi_2 > 0$ exists, and describe $\lim_{n\to\infty} P^n$ .

EXERCISE 19.6

One-step default from AAA: $p_{13}=0.01$ . Two-step: sum paths AAA $\to$ j $\to$ Default for $j\in\{1,2,3\}$ . Compare to one-step default from BBB ( $p_{23}=0.08$ ).

$P = \begin{pmatrix}0.92&0.07&0.01\\0.02&0.90&0.08\\0&0&1\end{pmatrix}$ .

One-step default from AAA: $p_{13} = 0.01 = 1\%$ .

Two-step default from AAA: $(P^2)_{13} = p_{11}p_{13}+p_{12}p_{23}+p_{13}p_{33}$ .

$= 0.92(0.01)+0.07(0.08)+0.01(1) = 0.0092+0.0056+0.01 = 0.0248 = 2.48\%$ .

The $0.0092$ is AAA $\to$ AAA $\to$ Default; $0.0056$ is AAA $\to$ BBB $\to$ Default; $0.01$ is AAA $\to$ Default $\to$ Default.

One-step default from BBB: $p_{23} = 0.08 = 8\%$ — eight times higher than AAA's one-step rate.

BBB is riskier at every horizon in this matrix — consistent with credit rating ordering.

Using $P = \begin{pmatrix}0.92&0.07&0.01\\0.02&0.90&0.08\\0&0&1\end{pmatrix}$ , compute the probability of default within 1 step and within 2 steps starting from AAA. Compare to the one-step default probability from BBB.

07 · Summary

Term	Definition
Stochastic matrix	$p_{ij} \geq 0$ ; each row sums to 1
Markov property	$P(X_{t+1}\mid X_t,\ldots) = P(X_{t+1}\mid X_t)$
$n$ -step distribution	$\boldsymbol{\mu}^{(n)} = \boldsymbol{\mu}^{(0)} P^n$
Stationary $\boldsymbol{\pi}$	$\boldsymbol{\pi}P=\boldsymbol{\pi}$ , $\sum\pi_i=1$
Eigenvector form	$P^T\boldsymbol{\pi}^T = \boldsymbol{\pi}^T$ — eigenvalue 1
Regular chain	$P^k > 0$ entrywise; $\boldsymbol{\mu}^{(t)} \to \boldsymbol{\pi}$
Absorbing state	$p_{ii}=1$ ; never left once entered
Credit application	Transition matrices for rating migration and default

Next: Numerical Stability & Conditioning — why LU, QR, and SVD behave differently in floating-point arithmetic, and how condition numbers quantify sensitivity in portfolio and risk computations.