# The occupancy problem

The occupancy problem is a classic problem in probability. The setting of the problem is that balls are randomly distributed into cells (or boxes or other containers) one at a time. After all the balls are distributed into the cells, we examine how many of the cells are occupied with balls. In this post, we examine the occupancy problem using Markov chains.

The Occupancy Problem

As indicated above, we consider the random experiment of randomly placing $k$ balls into $n$ cells. The following diagram shows the result of randomly distributing 8 balls into 6 cells (or boxes).

Figure 1 – Occupancy Problem – throwing balls into cells

One ball is thrown into the 6 boxes for a total of 8 balls. In Figure 1, a total of four cells are occupied. In the current discussion, the focus is on the number of occupied cells and not on which of the cells are occupied. Since Figure 1 has 6 cells, we can also view the throwing a ball into the 6 boxes as rolling a fair die (the occupied cell would be the face value that turns up). Thus we can view the experiment in Figure 1 a rolling a die 8 times. Then at the end, we look at the number faces that appear.

Figure 2 – Occupancy Problem – rolling die repeatedly and counting the faces that appear

The top of Figure 2 shows the 8 rolls of the die. Four faces turn up in these 8 rolls – faces 2, 4, 5 and 6, the same result described in Figure 1. Another way to look at the occupancy problem is that of random sampling. Each time a ball is thrown into $n$ cells, the action can be viewed as randomly selecting one of the $n$ cells. Thus the occupancy problem can be viewed as randomly selecting $k$ balls (one at a time with replacement) from an urn with $n$ balls labeled $1,2,\cdots,n$. The following diagram shows an urn with 6 balls labeled 1 through 6. Then Figure 1 and Figure 2 would be equivalent to randomly selecting 8 balls with replacement from this urn.

Figure 3 – Occupancy Problem – random selection of balls with replacement from an urn and counting the distinct numbers that are drawn

Figure 1 describes a generic setting of the occupancy problem – throwing balls at random into cells. The interpretation in Figure 2 is ideal for the setting of six cells. If the number of cells is not six, we can view the problem as rolling an $n$-sided die $k$ times. The random selection of balls with replacement (Figure 3) is also a good way to view the occupancy problem. Regardless of how we view the occupancy problem, we consider the following questions. In randomly assigning $k$ balls into $n$ cells,

• What is the probability that exactly $j$ cells are occupied with balls where $j=1,2,\cdots,n$?
• What is the expected number of occupied cells?

The first question is about the probability distribution of the number of occupied cells after the balls are thrown. The second question is about the mean of the probability distribution of the number of occupied cells. Once the probability distribution is known, we can calculate the distributional quantities such as the mean and variance. Thus the main focus is on the first question.

The occupancy problem has been discussed in a companion blog. In this blog post, the first question is answered by using the multinomial theorem (applied twice). Another blog post develops a formula for the probability distribution of the number of occupied cells. These previous blog posts use counting methods to solve the occupancy problem. For example, in throwing eight balls into 6 cells, there are $6^8$ many distinct outcomes. How many of these correspond to only one occupied cell, two occupied cells and so on? In this post, we present another way to answer the same question using Markov chain.

The occupancy problem discussed here and in the previous blog posts assumes that each ball is equally likely to be assigned into any one of the cells. Thus the occupancy problem where the weights of assignment of the cells are not uniform would be an interesting extension of the problem.

Applying Markov Chains

The notion is Markov chains is introduced here. The calculation involving transition probability matrix is discussed here.

To demonstrate, we use the specific example of randomly distributing balls into 6 cells. First, define a Markov chain. The state of the Markov chain is the number of occupied cells at any given time. Let $X_0$ denote the number of occupied cells at the beginning of the experiment (before any balls are thrown). If all cells are unoccupied at the beginning, then $X_0=0$. For $n \ge 1$, $X_n$ be the number of occupied cells after the $n$th ball has been distributed. The resulting stochastic process $\left\{X_n: n=0,1,2,\cdots \right\}$ is a Markov chain since the state $X_{n+1}$ only depends on the preceding state $X_n$. The following is the transition probability matrix.

$\displaystyle \mathbf{P} = \bordermatrix{ & 0 & 1 & 2 & 3 & 4 & 5 & 6 \cr 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 1 & 0 & \frac{1}{6} & \frac{5}{6} & 0 & 0 & 0 & 0 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 2 & 0 & 0 & \frac{2}{6} & \frac{4}{6} & 0 & 0 & 0 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 3 & 0 & 0 & 0 & \frac{3}{6} & \frac{3}{6} & 0 & 0 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 4 & 0 & 0 & 0 & 0 & \frac{4}{6} & \frac{2}{6} & 0 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 5 & 0 & 0 & 0 & 0 & 0 & \frac{5}{6} & \frac{1}{6} \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 6 & 0 & 0 & 0 & 0 & 0 & 0 & 1 \cr } \qquad$

The matrix $\mathbf{P}$ contains the one-step transition probabilities, i.e. the probabilities of the the next state given the current state. If the Markov chain is in state 0 (no occupied cells), the the next state is 1 for certain. If the current state is 1 (one occupied cell), then the next ball is is the occupied cell with probability 1/6 and is in one of the empty cells with probability 5/6. In general, if there are currently $i$ occupied cells, then the next ball is either in one of the $i$ occupied cells with probability $i/6$ (meaning the next state remains at $i$) or in one of the unoccupied cells with probability $(6-i)/6$ (meaning the next state is $i+1$). Since the number of occupied cells cannot decrease, the transition probabilities in $\mathbf{P}$ have the appearance of moving diagonally from the upper left corner to the lower right corner. State 6 (all cells are occupied) is a called an absorbing state. When the Markov chain is transitioned to state 6, it stays there for ever no matter how many more additional balls are thrown into the cells.

The transition probability matrix $\mathbf{P}$ holds the key to answering the first question indicated above. In throwing $k$ balls into 6 cells, the answer lies in the matrix $\mathbf{P}^k$, the $k$th power of $\mathbf{P}$ (the matrix obtained by multiplying $\mathbf{P}$ by itself $k$ times). This can be done by using software, e.g. using an online matrix calculator.

Consider the example of throwing eight balls into 6 cells. The following is the matrix $\mathbf{P}^8$.

$\displaystyle \mathbf{P}^8 = \bordermatrix{ & 0 & 1 & 2 & 3 & 4 & 5 & 6 \cr 0 & 0 & 0.00000357 & 0.00226838 & 0.06901578 & 0.36458333 & 0.45010288 & 0.11402606 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 1 & 0 & 0.0000006 & 0.0007591 & 0.03602014 & 0.27756344 & 0.49661351 & 0.18904321 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 2 & 0 & 0 & 0.00015242 & 0.01501534 & 0.18815015 & 0.50831619 & 0.28836591 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 3 & 0 & 0 & 0 & 0.00390625 & 0.10533658 & 0.47531221 & 0.41544496 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 4 & 0 & 0 & 0 & 0 & 0.03901844 & 0.38709919 & 0.57388236 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 5 & 0 & 0 & 0 & 0 & 0 & 0.23256804 & 0.76743196 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 6 & 0 & 0 & 0 & 0 & 0 & 0 & 1 \cr } \qquad$

Each entry in $\mathbf{P}^8$ is denoted by $P_{ij}^8$, the 8-step transition probability, located in the row corresponding to the state $i$ and in the column corresponding to the state $j$. For example, $P_{03}^8=0.069$, indicating that there is a 6.9% chance that 3 cells are occupied after eight balls are thrown (with all cells being empty at the beginning). Another example: $P_{13}^8=0.036$. So if there is one occupied cell at the beginning, there is a 3.6% chance that 3 cells are occupied after throwing 8 balls into the 6 cells. So each row in $\mathbf{P}^8$ gives a probability distribution of the number of occupied cells depending the initial state.

The first question stated above assumes that the all cells are empty at the beginning. Thus the first row of the matrix $\mathbf{P}^8$ gives the answers.

$P[\text{1 occupied cell}]=P_{01}^8=0.00000357$

$P[\text{2 occupied cell}]=P_{02}^8=0.00226838$

$P[\text{3 occupied cell}]=P_{03}^8=0.06901578$

$P[\text{4 occupied cell}]=P_{04}^8=0.36458333$

$P[\text{5 occupied cell}]=P_{05}^8=0.45010288$

$P[\text{6 occupied cell}]=P_{06}^8=0.11402606$

In randomly throwing 8 balls into 6 cells, the more likely outcomes would be having 4, 5 or 6 occupied cells with the mostly likely being 4 occupied cells. Let $Y$ be the number of occupied cells after 8 balls are thrown into 6 cells. The mean number of occupied cells is:

\displaystyle \begin{aligned} E[Y]&=1 \times 0.00000357+2 \times 0.00226838+3 \times 0.06901578 \\&\ \ + 4 \times 0.36458333+5 \times 0.45010288+6 \times 0.11402606 \\&=4.60459175 \end{aligned}

To give an indication that the probability distribution is correct, we simulate 8 rolls of a die 1,000 times using the =RANDBETWEEN(1, 6) function in Excel. The results: 0% (1 occupied cell), 0.4% (2 occupied cells), 7.3% (3 occupied cells), 35.2% (4 occupied cells), 47.3% (5 occupied cells) and 9.8% (6 occupied cells). These results are in general agreement with the calculated distribution. Of course, the larger the simulation, the closer the simulated results will be to the theoretical distribution.

Here’s the algorithm for solving the occupancy problem using the Markov chain approach.

If the occupancy problem involves throwing balls into $n$ cells, the state of the Markov chain is the number of occupied cells at any given time. Set up the one-step transition probability matrix $\mathbf{P}$ as follows. Set $P_{01}=1$ and $P_{n,n-1}=1$. For $0, set $P_{ii}=i/n$, $P_{i,i+1}=(n-i)/n$. Then the matrix $\mathbf{P}^k$ gives the probability distribution of the number of occupied cells after $k$ balls are randomly distributed into the $n$ cells. The row in $\mathbf{P}^k$ corresponding to the state $i$ gives the probability of the number of occupied cells when there are $i$ occupied cells initially. If initially all cells are empty, then the first row of $\mathbf{P}^k$ gives the probability distribution of the number of occupied cells.

We would like to make further comment on the example. Assuming that all cells are empty at the beginning, the example can be worked by eliminating the first row of $\mathbf{P}$, the row for state 0. The first ball will always go into one empty cell. Thus in throwing 8 balls, we can focus on 7 balls going into 6 cells with one cell already occupied (by the first ball). Thus we can focus on the following transition probability matrix.

$\displaystyle \mathbf{P} = \bordermatrix{ & 1 & 2 & 3 & 4 & 5 & 6 \cr 1 & \frac{1}{6} & \frac{5}{6} & 0 & 0 & 0 & 0 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 2 & 0 & \frac{2}{6} & \frac{4}{6} & 0 & 0 & 0 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 3 & 0 & 0 & \frac{3}{6} & \frac{3}{6} & 0 & 0 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 4 & 0 & 0 & 0 & \frac{4}{6} & \frac{2}{6} & 0 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 5 & 0 & 0 & 0 & 0 & \frac{5}{6} & \frac{1}{6} \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 6 & 0 & 0 & 0 & 0 & 0 & 1 \cr } \qquad$

Assuming that the first ball goes into one of the empty cells, we focus on the next 7 balls. So we compute the matrix $\mathbf{P}^7$.

$\displaystyle \mathbf{P}^7 = \bordermatrix{ & 1 & 2 & 3 & 4 & 5 & 6 \cr 1 & 0.00000357 & 0.00226838 & 0.06901578 & 0.36458333 & 0.45010288 & 0.11402606 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 2 & 0 & 0.00045725 & 0.02942101 & 0.26015947 & 0.50591564 & 0.20404664 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 3 & 0 & 0 & 0.0078125 & 0.15214549 & 0.50951646 & 0.33052555 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 4 & 0 & 0 & 0 & 0.05852766 & 0.44110797 & 0.50036437 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 5 & 0 & 0 & 0 & 0 & 0.27908165 & 0.72091835 \cr \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } & \text{ } \cr 6 & 0 & 0 & 0 & 0 & 0 & 1 \cr } \qquad$

Note that the first row of $\mathbf{P}^7$ is identical to the first row of $\mathbf{P}^8$ earlier. So a slightly different algorithm is to put one ball in a cell and then focus on the random behavior of the next $k-1$ balls. In other words, set up the transition probability matrix $\mathbf{P}$ with only the states $1, 2,\cdots,n$ and then raise $\mathbf{P}$ to $k-1$.

Remarks

As the above example demonstrates, the occupancy problem is a matter of calculating the power of a transition probability matrix. It is an algorithm that is suitable for computer implementation. It illustrates the versatility of the Markov chain method. The combinatorial approach of using multinomial theorem (discussed here and here) is also a valuable approach. Each approach provides its own unique insight. The post ends with an exercise.

Exercise
Suppose that eleven candies are randomly distributed to 4 children. Suppose that the names of the 4 children are Marcus, Issac, Samantha and Paul.

• Given the one-step transition probability matrix for the Markov chain describing the process of distributing candies to 4 children.
• What is the probability distribution of the number of children who have received candies?
• What is the probability all 4 children receiving candies?
• What is the mean number of children who have received candies?
• What is the probability that Marcus receives 1 candy, Issac receives 4 candies, Samantha receives 4 candies and Paul receives 2 candies?
• What is the probability that one of the children receives 1 candy, another child receives 4 candies, another child receives 4 candies and the last child receives 2 candies?

$\text{ }$

$\text{ }$

$\text{ }$

$\copyright$ 2017 – Dan Ma

Advertisements

# A first look at applications of Markov chains

This post presents several interesting examples of Markov chain models. Previous posts on Markov chains are: an introduction, how to work with transition probabilities (here and here).

Examples

The following gives several examples of Markov chains, discussed at an introductory level. They will be further developed in subsequent posts.

Example 1 – The Ehrenfest Chain

The Ehrenfest chain, named for the physicist Paul Ehrenfest, is a simple, discrete model for the exchange of gas molecules contained in a volume divided into two regions A and B by a permeable membrane. Suppose that the gas has $N$ molecules. At each period of time, a molecule is chosen at random from the set of $N$ molecules and moved from the region that it is in to the other region.

Figure 1 – The Efrenfest Chain

The model has a simple mathematical description using balls and urns. Suppose that two urns, labeled A and B, contain $N$ balls, labeled $1, 2, 3, \cdots, N$. At each step (i.e. at each discrete time unit), an integer is selected at random from $1, 2, 3, \cdots, N$ independent of the past selections. Then the ball labeled by the selected integer is removed from its urn and placed in the other urn. The procedure is repeated indefinitely.

The state of the system at time $n=0,1,2,\cdots$ is the number of balls in urn A and is denoted by $X_n$. Then $\left\{X_n: n=0,1,2,\cdots \right\}$ is a Markov chain on the state space $\left\{0,1,\cdots,N \right\}$.

When the process is in state 0 (urn A is empty), the next state is 1. When the process is in state $N$ (urn A is full), the next state is $N-1$. When the process is in state $i$ where $0, the next state of the process is either $i-1$ or $i+1$. The following gives the one-step transition probabilities:

$\displaystyle P_{ij} = \left\{ \begin{array}{ll} \displaystyle 1 &\ \ \ \ \ \ i=0 \text{ and } j=1 \\ \text{ } & \text{ } \\ \displaystyle 1 &\ \ \ \ \ \ i=N \text{ and } j=N-1 \\ \text{ } & \text{ } \\ \displaystyle \frac{i}{N} &\ \ \ \ \ \ 0

When urn A has $i$ balls, there is a probability of $\frac{i}{N}$ such that the randomly selected ball is from urn A and thus urn A loses a ball. On the other hand, there is a probability of $1-\frac{i}{N}$ such that the randomly selected ball is from urn B and thus urn A gains a ball. As an illustration, the following gives the transition probability matrix for $N=5$ balls.

$\mathbf{P} = \bordermatrix{ & 0 & 1 & 2 & 3 & 4 & 5 \cr 0 & 0 & 1 & 0 & 0 & 0 & 0 \cr 1 & \frac{1}{5} & 0 & \frac{4}{5} & 0 & 0 & 0 \cr 2 & 0 & \frac{2}{5} & 0 & \frac{3}{5} & 0 & 0 \cr 3 & 0 & 0 & \frac{3}{5} & 0 & \frac{2}{5} & 0 \cr 4 & 0 & 0 & 0 & \frac{4}{5} & 0 & \frac{1}{5} \cr 5 & 0 & 0 & 0 & 0 & 1 & 0 \cr } \qquad$

An important problem in the Ehrenfest chain is the long-term, or equilibrium, distribution of $X_n$, the number of balls in urn A. Another interesting problem concerns the changes in the composition of the two regions over time. For example, if all molecules are in one region at the beginning, how long on average will it be before each region has half of the molecules? The theory of Markov chains provide a good method for answering such questions.

Example 2 – Random Walk

The simplest random walk is a Markov chain $\left\{X_n: n=0,1,2,\cdots \right\}$ such that each state is the result of a random one-unit up or down move from the previous state.

Figure 2 – Five Simulations of a Random Walk

In the random walk in Figure 1, each state is one unit above or below the preceding state with equal probability. Instead of a random one-unit up or down move, let’s give a more general description of a random walk. The moves in the random walk are determined by a predetermined discrete distribution. Let $Y_1, Y_2, Y_3 \cdots$ be integer-valued random variables that are independent and identically distributed. Let $P[Y=y]$ be the common probability function. Let $X_0$ be an integer-valued random variable that is independent of the random variables $Y_n$. The random variable $X_0$ will be the initial position of the random walk. The random variables $Y_n$ are the increments (they are the amounts added to the stochastic process as time increases).

For $n \ge 1$, define $X_n=X_0+Y_1+\cdots+Y_{n-1}+Y_n$. Then $\left\{X_n: n=0,1,2,\cdots \right\}$ is called a random walk. It is a Markov chain with the state space the set of all integers. Wherever the process is at any given time, the next step in the process is always a walk in a distance and direction that is dictated by the independent random variables $Y_n$. If the random variables $Y_n$ are independent Bernoulli variables with $P(Y_n=1)=p$ and $P(Y_n=-1)=1-p$, then we revert back to the simple random walks described in Figure 1.

To transition from state $i$ to state $j$, the increment $Y_n$ must be $j-i$. Thus the one-step transition probabilities $P_{ij}$ are defined by $P_{ij}=P[Y=j-i]$. Recall that $P[Y=y]$ is the common probability function for the sequence $Y_1, Y_2, Y_3 \cdots$. For a more detailed description, see this previous post.

Example 3 – Gambler’s Ruin

A gambler’s ruin is a simple random walk that can take on only finitely many states. At each time period, the state is one-unit higher or lower than the preceding state as a result of a random bet. More specifically, suppose that a gambler starts out with $d$ units in capital (in dollars or other monetary units) and makes a series of one-unit bets against the house. For the gambler, the probabilities of winning and losing each bet are $p$ and $1-p$, respectively. Whenever the capital reaches zero, the gambler is in ruin and his capital remains zero thereafter. On the other hand, if the capital of the gambler increases to $m$ where $d, then the gambler quits playing. Let $X_n$ be the capital of the gambler after the $n$th bet. Then $\left\{X_n: n=0,1,2,3,\cdots \right\}$ is a random walk. The starting state is $X_0=d$. The states 0 and $m$ are absorbing states. The following gives the transition probabilities.

$\displaystyle P_{ij} = \left\{ \begin{array}{ll} \displaystyle p &\ \ \ \ \ \ j=i+1,i \ne 0, i \ne m \\ \text{ } & \text{ } \\ \displaystyle 1-p &\ \ \ \ \ \ j=i-1,i \ne 0, i \ne m \\ \text{ } & \text{ } \\ \displaystyle 1 &\ \ \ \ \ \ i=0,j=0 \\ \text{ } & \text{ } \\ \displaystyle 1 &\ \ \ \ \ \ i=m,j=m \\ \text{ } & \text{ } \\ 0 &\ \ \ \ \ \ \text{otherwise} \end{array} \right.$

Whenever the gambler reaches the absorbing state of 0, the gambler is in ruin. If the process reaches the absorbing state of $m$, the gambler strikes gold. If the initial capital of the gambler is small relative to the casino, there is a virtually 100% chance that the gambler will be in ruin even if each bet has even odds (see here for the calculation). One interesting questions: on average how long does it take for the gambler to be in ruin? Such questions will be answered after necessary tools are built in subsequent posts.

An alternative interpretation of the gamblerâ€™s ruin random walk is this. Assume that two players make a series of one-unit bets against each other and that the total capital between the two players is $m$ units with the first player having $d$ units in capital. Further suppose that the first player has probability $p$ of winning a bet and the second player has probability $1-p$ of winning a bet. Then the two gamblers play until one of them goes broke (is in ruin). Let $X_n$ be the capital of the first player after the $n$th play. Then $\left\{X_n: n=0,1,2,3,\cdots \right\}$ is a Markov chain that is a gambler’s ruin. The following graph shows five simulations of a gambler’s ruin from this previous post.

Figure 3 – Five Simulations of a Gambler’s Ruin

Example 4 – Discrete Birth and Death Chain

In a birth and death chain, the current state $i$ in the process is transitioned to $i+1$ (a birth), $i-1$ (a death) or $i$. The state space is either $\left\{0,1,2,3,\cdots \right\}$ or $\left\{0,1,2,3,\cdots, m \right\}$. The following gives the one-step transition probabilities.

$\displaystyle P_{ij} = \left\{ \begin{array}{ll} \displaystyle q_i &\ \ \ \ \ \ j=i-1 \\ \text{ } & \text{ } \\ \displaystyle r_i &\ \ \ \ \ \ j=i \\ \text{ } & \text{ } \\ \displaystyle p_i &\ \ \ \ \ \ j=i+1 \\ \text{ } & \text{ } \\ 0 &\ \ \ \ \ \ \text{otherwise} \end{array} \right.$

where for each state $i$, $q_i$ (the probability of down), $r_i$ (the probability of same) and $p_i$ (the probability of up) are non-negative real numbers such that $q_i+r_i+p_i=1$. The following gives the transition probability matrix for an illustrative case of $m=5$.

$\mathbf{P} = \bordermatrix{ & 0 & 1 & 2 & 3 & 4 & 5 \cr 0 & r_0 & p_0 & 0 & 0 & 0 & 0 \cr 1 & q_1 & r_1 & p_1 & 0 & 0 & 0 \cr 2 & 0 & q_2 & r_2 & p_2 & 0 & 0 \cr 3 & 0 & 0 & q_3 & r_3 & p_3 & 0 \cr 4 & 0 & 0 & 0 & q_4 & r_4 & p_4 \cr 5 & 0 & 0 & 0 & 0 & q_5 & r_5 \cr } \qquad$

In the above matrix, $q_0=0$ and $p_5=0$. If $r_0=1$, then state 0 is absorbing. If $p_0=1$, then state 0 is reflecting. On the other hand, if $r_5=1$, then state 5 is absorbing. If $q_5=1$, then state 5 is reflecting.

The Ehrenfest chain is an example of a birth and death chain (with the boundary states being reflecting). When the probability of down $q_i$, the probability of same $r_i$ and the probability of up $p_i$ do not vary according to the current state (i.e. they are constant), the resulting birth and death chain is a random walk. Thus the gambler’s ruin chain is also an example of a birth and death chain. The birth and death chain is suitable model for applications in which the state of the process is the population of a living system.

Example 5 – Discrete Queueing Markov Chain

Though continuous-time queueing models are much more realistic, a simple discrete-time queueing model is introduced here in order to illustrate applications of discrete Markov chains. Suppose that time is measured in a convenient time interval such as one minute or some appropriate group of minutes. The model assumes that during one period of time, only one customer is served if at least one customer is present. The model also assumes that a random number of new customers arrive at any given time period and that the numbers of customers arriving in the time periods form an independent identically distributed sequence of random variables.

More specifically, let $Y_1,Y_2,Y_3,\cdots$ be independent and identically distributed random variables that take on non-negative integers. Let $P[Y=k]$, $k=0,1,2,\cdots$, be the common probability function. The random variable $Y_n$ is the number of customers that arrive into the system during the $n$th period. Let $X_0$ be the number of customers present initially in the system. For $n \ge 1$, let $X_n$ be the number of customers in the system during the $n$th period. If the current state is $X_n$, then the next state is:

$\displaystyle X_{n+1} = \left\{ \begin{array}{ll} \displaystyle Y_{n+1} &\ \ \ \ \ \ X_n=0 \\ \text{ } & \text{ } \\ \displaystyle (X_n-1)+Y_{n+1} &\ \ \ \ \ \ X_n>0 \\ \end{array} \right.$

where $n \ge 0$. In essence, the number of customers in the system at any given time period is the number of new customers arriving plus the customers already in the system less one (if the system is not empty). If the system is empty, the number of customers is simply the number of new customers. It follows that $\left\{X_n: n \ge 0 \right\}$ is a Markov chain with the state space $\left\{0,1,2,\cdots \right\}$, the set of all non-negative integers. The one-step transition probabilities $P_{ij}$ are given by:

$\displaystyle P_{ij} = \left\{ \begin{array}{ll} \displaystyle P[Y=j] &\ \ \ \ \ \ i=0 \text{ and } j \in \left\{0,1,2,\cdots \right\} \\ \text{ } & \text{ } \\ \displaystyle P[Y=j-(i-1)] &\ \ \ \ \ \ i>0 \text{ and } j \in \left\{i-1,i,i+1,\cdots \right\} \\ \end{array} \right.$

where $P[Y=k]$ is the common probability function for the independent numbers of arrivals of customers $Y_1,Y_2,Y_3,\cdots$. If the $Y_n$ have a common binomial distributions with parameters 3 and $p=0.5$. Then the following gives the transition probability matrix.

$\displaystyle \mathbf{P} = \bordermatrix{ & 0 & \text{ } & 1 & \text{ } & 2 & \text{ } & 3 & \text{ } & 4 & \text{ } & 5 & \text{ } & 6 & \text{ } & 7 & \text{ } & 8 & \cdots \cr 0 & 0.125 & \text{ } & 0.375 & \text{ } & 0.375 & \text{ } & 0.125 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \cdots \cr 1 & 0.125 & \text{ } & 0.375 & \text{ } & 0.375 & \text{ } & 0.125 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \cdots \cr 2 & 0 & \text{ } & 0.125 & \text{ } & 0.375 & \text{ } & 0.375 & \text{ } & 0.125 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \cdots \cr 3 & 0 & \text{ } & 0 & \text{ } & 0.125 & \text{ } & 0.375 & \text{ } & 0.375 & \text{ } & 0.125 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \cdots \cr 4 & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0.125 & \text{ } & 0.375 & \text{ } & 0.375 & \text{ } & 0.125 & \text{ } & 0 & \text{ } & 0 & \cdots \cr 5 & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0.125 & \text{ } & 0.375 & \text{ } & 0.375 & \text{ } & 0.125 & \text{ } & 0 & \cdots \cr 6 & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0.125 & \text{ } & 0.375 & \text{ } & 0.375 & \text{ } & 0.125 & \cdots \cr 7 & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0.125 & \text{ } & 0.375 & \text{ } & 0.375 & \cdots \cr 8 & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0 & \text{ } & 0.125 & \text{ } & 0.375 & \cdots \cr \vdots & \vdots & \text{ } & \vdots & \text{ } & \vdots & \text{ } & \vdots & \text{ } & \vdots & \text{ } & \vdots & \text{ } & \vdots & \text{ } & \vdots & \text{ } & \vdots & \text{ } \cr } \qquad$

If the customer arrivals have a common Poisson distribution with parameter $\lambda=0.2$ (per period), then the following gives the transition probabilities.

$\displaystyle P_{ij} = \left\{ \begin{array}{ll} \displaystyle \frac{e^{-0.2} \ 0.2^j}{j!} &\ \ \ \ \ \ i=0 \text{ and } j \in \left\{0,1,2,\cdots \right\} \\ \text{ } & \text{ } \\ \displaystyle \frac{e^{-0.2} \ 0.2^{j-(i-1)}}{[j-(i-1)]!} &\ \ \ \ \ \ i>0 \text{ and } j \in \left\{i-1,i,i+1,\cdots \right\} \\ \end{array} \right.$

Example 6 – Discrete Branching Chain

Suppose that we have a system of particles that can generate new particles of the same type, e.g. particles such as neutrons and bacteria.

In such a system, we assume that a particle is replaced by a ransom number of new particles (regarded as offspring of the original particle). Furthermore, we assume that each particle generates offspring independently and offspring generation follows the same distribution. Let $Y$ be the random variable that is the common distribution for offspring generation across all particles in the system. Let $P[Y=k]$ be the common probability function of the number of offspring of a particle.

For $n \ge 0$, let $X_n$ be the number of particles in the $n$th generation in the system. It follows from the assumptions that with $X_n=i$

$\displaystyle X_{n+1}=\sum \limits_{k=1}^{i} Y_k$

where $Y_1,Y_2, \cdots, Y_i$ are independent random variables with common probability function $P[Y=k]$. Then $\left\{X_n: n \ge 0 \right\}$ is a Markov chain with the state space the set of all non-negative integers. The one-step transition probabilities $P_{ij}$ are

$\displaystyle P_{ij}=P \biggl[\sum \limits_{k=1}^{i} Y_k = j \biggr]$

In other words, the transition probability $P_{ij}$ is defined by the probability that the independent sum $Y_1+\cdots+Y_i$ taking on the next state $j$.

Note that state 0 is an absorbing state, which means extinction. An interesting problems in branching chains is the computation of the probability of extinction. One approach is to compute the probability of extinction for a branching chain with a single particle, i.e. the probability $\rho$ of starting in state 1 and being absorbed into state 0. Then the probability of extinction for a branching chain starting with $k$ particles is then $\rho^k$ since the particles are assumed to generate new particles independently.

$\text{ }$

$\text{ }$

$\text{ }$

$\copyright$ 2017 – Dan Ma

# Chapman-Kolmogorov Equations

Stochastic processes and Markov chains are introduced in this previous post. Transition probabilities are an integral part of the theory of Markov chains. The post preceding this one is a beginning look at transition probabilities. This post shows how to calculate the $n$-step transition probabilities. The Chapman-Kolmogorov equations are also discussed and derived.

Introductory Example

Suppose $\left\{X_n: n=0,1,2,\cdots \right\}$ is a Markov chain with the transition probability matrix $\mathbf{P}$. The entries of the matrix are the one-step transition probabilities $P_{ij}$. The number $P_{ij}$ is the probability that the Markov chain will move to state $j$ at time $m+1$ given that it is in state $i$ at time $m$, independent of where the chain was prior to time $m$. Thus $P_{ij}$ can be expressed as the following conditional probability.

$(1) \ \ \ \ \ P_{ij}=P[X_{m+1}=j \lvert X_m=i]$

Thus the future state only depends on the period immediately preceding it. This is called the Markov property. Also note that $P_{ij}$ is independent of the time period $m$. Any Markov chain with this property is called a time-homogeneous Markov chain or stationary Markov chain.

The focus of this post is to show how to calculate the probability that the Markov chain will move to state $j$ at time $n+m$ given that it is in state $i$ at time $m$. This probability is denoted by $P_{ij}^n$. In other words, $P_{ij}^n$ is the probability that the chain will be in state $j$ after the Markov chain goes through $n$ more periods given that it is in state $i$ in the current period. The probability $P_{ij}^n$, as a conditional probability, can be notated as follows:

$(2) \ \ \ \ \ P_{ij}^n=P[X_{m+n}=j \lvert X_m=i]$

In (2), the $n$ in the $n$-step transition probabilities satisfies $n \ge 0$. Note that when $n=0$, $P_{ij}^0=1$ for $i=j$ and $P_{ij}^0=0$ for $i \ne j$. Including the case for $n=0$ will make the Chapman-Kolmogorov equations work better.

Before discussing the general method, we use examples to illustrate how to compute 2-step and 3-step transition probabilities. Consider a Markov chain with the following transition probability matrix.

$\mathbf{P} = \bordermatrix{ & 0 & 1 & 2 \cr 0 & 0.6 & 0.2 & 0.2 \cr 1 & 0.3 & 0.5 & 0.2 \cr 2 & 0.4 & 0.3 & 0.3 \cr } \qquad$

Calculate the two-step transition probabilities $P_{02}^2$, $P_{12}^2$ and $P_{22}^2$. Then calculate the three-step transition probability $P_{02}^3$ using the two-step transition probabilities.

First, let’s handle $P_{02}^2$. We can condition on the first steps. To go from state 0 to state 2 in two steps, the chain must first go to an interim state and then from that state go to state 2.

\displaystyle \begin{aligned}P_{02}^2&=\sum \limits_{k=0}^2 P_{0k} \ P_{k2} \\&=P_{00} \ P_{02}+P_{01} \ P_{12}+P_{02} \ P_{22} \\&=0.6 \times 0.2+0.2 \times 0.2+0.2 \times 0.3 \\&=0.22 \end{aligned}

Note that the above calculation lists out the three possible paths to go from state 0 to state 2 in two steps – from state 0 to state 0 and then from state 0 to state 2, from state 0 to state 1 and then from state 1 to state 2 and from state 0 to state 2 and then from state 2 to state 2. Looking at the calculation more closely, the above calculation is basically the first row of $\mathbf{P}$ (the row corresponding to state 0) multiplying the third column of $\mathbf{P}$ (the column corresponding to state 2).

$P_{02}^2= \left[\begin{array}{ccc} 0.6 & 0.2 & 0.2 \\ \end{array}\right] \left[\begin{array}{c} 0.2 \\ 0.2 \\ 0.3 \end{array}\right]= \left[\begin{array}{c} 0.22 \end{array}\right]$

By the same idea, the following gives the other two 2-step transition probabilities.

\displaystyle \begin{aligned}P_{12}^2&=\sum \limits_{k=0}^2 P_{1k} \ P_{k2} \\&=P_{10} \ P_{02}+P_{11} \ P_{12}+P_{12} \ P_{22} \\&=0.3 \times 0.2+0.5 \times 0.2+0.2 \times 0.3 \\&=0.22 \end{aligned}

\displaystyle \begin{aligned}P_{22}^2&=\sum \limits_{k=0}^2 P_{2k} \ P_{k2} \\&=P_{20} \ P_{02}+P_{21} \ P_{12}+P_{22} \ P_{22} \\&=0.4 \times 0.2+0.3 \times 0.2+0.3 \times 0.3 \\&=0.23 \end{aligned}

As discussed, the above two calculations can be viewed as the sum of all the possible paths to go from the beginning state to the end state (conditioning on the interim state in the middle) or as a row in the transition probability $\mathbf{P}$ multiplying a column in $\mathbf{P}$. The following shows all three calculations in terms of matrix calculation.

$P_{02}^2= \left[\begin{array}{ccc} 0.6 & 0.2 & 0.2 \\ \end{array}\right] \left[\begin{array}{c} 0.2 \\ 0.2 \\ 0.3 \end{array}\right]= \left[\begin{array}{c} 0.22 \end{array}\right]$

$P_{12}^2= \left[\begin{array}{ccc} 0.3 & 0.5 & 0.2 \\ \end{array}\right] \left[\begin{array}{c} 0.2 \\ 0.2 \\ 0.3 \end{array}\right]= \left[\begin{array}{c} 0.22 \end{array}\right]$

$P_{22}^2= \left[\begin{array}{ccc} 0.4 & 0.3 & 0.3 \\ \end{array}\right] \left[\begin{array}{c} 0.2 \\ 0.2 \\ 0.3 \end{array}\right]= \left[\begin{array}{c} 0.23 \end{array}\right]$

The view of matrix calculation will be crucial in understanding Chpan-Kolmogorov equations discussed below. To conclude the example, consider the 3-step transition probability $P_{02}^3$. We can also condition on the first step. The chain goes from state 0 to the next step (3 possibilities) and then goes from that state to state 2 in two steps.

\displaystyle \begin{aligned} P_{02}^3&=\sum \limits_{k=0}^2 P_{0k} \ P_{k2}^2 \\&=P_{00} \ P_{02}^2+P_{01} \ P_{12}^2+P_{02} \ P_{22}^2 \\&=0.6 \times 0.22+0.2 \times 0.22+0.2 \times 0.23 \\&=0.222 \end{aligned}

The example shows that the calculation of a 3-step transition probability is based 2-step probabilities. Thus longer length transition probabilities can be built up from shorter length transition probabilities. However, it pays to focus on a more general framework before carrying out more calculation. The view of matrix calculation demonstrated above will help in understanding the general frame work.

Chapman-Kolmogorov Equations

The examples indicate that finding $n$-step transition probabilities involve matrix calculation. Let $\mathbf{Q}_n$ be the $n$-step transition probability matrix. The goal now is to have a systematic way to compute the entries in the matrix $\mathbf{Q}_n$. The computation is based on the Chapman-Kolmogorov equations. These equations are:

$\displaystyle (3) \ \ \ \ \ P_{ij}^{n+m}=\sum \limits_{k=0}^\infty P_{ik}^n \times P_{kj}^m$

for $n, m \ge 0$ and for all $i,j$. For a finite-state Markov chain, the summation in (3) does not go up to infinity and would have an upper limit. The number $P_{ij}^{n+m}$ is the probability that the chain will be in state $j$ after taking $n+m$ steps if it is currently in state $i$. The following gives the derivation of (3).

\displaystyle \begin{aligned} P_{ij}^{n+m}&=P[X_{n+m}=j \lvert X_0=i] \\&=\sum \limits_{k=0}^\infty P[X_{n+m}=j, X_n=k \lvert X_0=i]\\&=\sum \limits_{k=0}^\infty \frac{P[X_{n+m}=j, X_n=k, X_0=i]}{P[X_0=i]} \\&=\sum \limits_{k=0}^\infty \frac{P[X_n=k,X_0=i]}{P[X_0=i]} \times \frac{P[X_{n+m}=j, X_n=k, X_0=i]}{P[X_n=k,X_0=i]} \\&=\sum \limits_{k=0}^\infty P[X_n=k \lvert X_0=i] \times P[X_{n+m}=j \lvert X_n=k, X_0=i] \\&=\sum \limits_{k=0}^\infty P[X_n=k \lvert X_0=i] \times P[X_{n+m}=j \lvert X_n=k] \ * \\&=\sum \limits_{k=0}^\infty P_{ik}^n \times P_{kj}^m \end{aligned}

Here’s the idea behind the derivation. The path from state $i$ to state $j$ in $n+m$ steps can be broken down into two paths, one from state $i$ to an intermediate state $k$ in the first $n$ transitions, and the other from state $k$ to state $j$ in the remaining $m$ transitions. Summing over all intermediate states $k$ yields the probability that the process will go from state $i$ to state $j$ in $n+m$ transitions.

The entries in the matrix $\mathbf{Q}_n$ can be then computed by (3). There is an interesting an useful interpretation of (3). The following is another way to state the Chapman-Kolmogorov equations:

$(4) \ \ \ \ \ \mathbf{Q}_{n+m}=\mathbf{Q}_n \times \mathbf{Q}_m$.

A typical element of the matrix $\mathbf{Q}_n$ is $P_{ik}^n$ and a typical element of the matrix $\mathbf{Q}_m$ is $P_{kj}^m$. Note that $P_{ik}^n$ with $k$ varying is a row in $\mathbf{Q}_n$ (the row corresponding to the state $i$) and that $P_{kj}^n$ with $k$ varying is a column in $\mathbf{Q}_m$ (the column corresponding to the state $j$).

$\left[\begin{array}{cccccc} P_{i0}^n & P_{i1}^n & \cdots & P_{ik}^n & \cdots & P_{iw}^n \\ \end{array}\right] \left[\begin{array}{c} P_{0j}^m \\ \text{ } \\ P_{1j}^m \\ \vdots \\ P_{kj}^m \\ \vdots \\ P_{wj}^m \end{array}\right]$

The product of the above row and column is the transition probability $P_{ij}^{n+m}$.

Powers of the One-Step Transition Probability Matrix

Let $\mathbf{P}$ be the one-step transition probability matrix of a Markov chain. Let $\mathbf{Q}_n$ be the $n$-step transition probability matrix, which can be computed by using the Chapman-Kolmogorov equations. We now derive another important fact.

The $n$-step transition probability matrix $\mathbf{Q}_n$ is obtained by multiplying the one-step transition probability matrix $\mathbf{P}$ by itself $n$ times, i.e. $\mathbf{Q}_n$ is the $n$th power of $\mathbf{P}$.

This fact is important in terms of calculation of transition probabilities. Compute $\mathbf{P}^n$, the $n$th power of $\mathbf{P}$ (in terms of matrix calculation). Then $P_{ij}^n$ is simply an entry in $\mathbf{P}^n$ (in the row that corresponds to state $i$ and in the column that corresponds to state $j$). If the matrix size is large and/or $n$ is large, the matrix multiplication can be done using software or by using an online matrix calculator (here’s one matrix calculator).

Of course, the above fact is also important Markov chain theory in general. Almost all mathematical properties of Markov chains are at root based on this basic fact.

We can establish this basic fact using an induction argument. Clearly $\mathbf{Q}_1=\mathbf{P}^1=\mathbf{P}$. Suppose that the fact is true for $n-1$. Based on (4), $\mathbf{Q}_{n}=\mathbf{Q}_{n-1} \times \mathbf{Q}_1$. Continuing the derivation: $\mathbf{Q}_{n}=\mathbf{Q}_{n-1} \times \mathbf{Q}_1=\mathbf{P}^{n-1} \times \mathbf{P}=\mathbf{P}^n$.

The Row Vector and the Column Vector

As demonstrated in the preceding section, $\mathbf{P}^n$, the $n$th power of the $\mathbf{P}$, is precisely the $n$-step transition probability matrix. Let’s examine the matrix $\mathbf{P}^n$ more closely. Suppose that the Markov chain has a state space $\left\{0,1,2,\cdots,w \right\}$. The following shows the matrix $\mathbf{P}^n$ with special focus on a generic row and a generic column.

$\mathbf{P}^n= \left[\begin{array}{ccccccc} P_{0,0}^n & P_{0,1}^n & \cdots & P_{0,k}^n & \cdots & P_{0,w}^n \\ P_{1,0}^n & P_{1,1}^n & \cdots & P_{1,k}^n & \cdots & P_{1,w}^n \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\ P_{i,0}^n & P_{i,1}^n & \cdots & P_{i,k}^n & \cdots & P_{i,w}^n \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\ P_{w,0}^n & P_{w,1}^n & \cdots & P_{w,k}^n & \cdots & P_{w,w}^n \\ \end{array}\right]$

Now look at the row corresponding to the state $i$ and call it $R_i$. Also look at the column corresponding to the state $k$ and call it $C_k$. They are separated from the matrix $\mathbf{P}^n$ below.

$R_i=\left[\begin{array}{cccccc} P_{i,0}^n & P_{i,1}^n & \cdots & P_{i,k}^n & \cdots & P_{i,w}^n \\ \end{array}\right] \ \ \ \ \ \text{row vector}$

$C_k=\left[\begin{array}{c} P_{0,k}^n \\ \text{ } \\ P_{1,k}^n \\ \vdots \\ P_{i,k}^n \\ \vdots \\ P_{w,k}^n \end{array}\right] \ \ \ \ \ \text{column vector}$

The row vector $R_i$ is a conditional distribution. It gives the probabilities of the process transitioning (after $n$ transitions) into one of the states given the process starts in state $i$. If it is certain that the process begins in state $i$, then $R_i$ gives the probability function for the random variable $X_n$ since $P[X_n=k]=P_{i,k}^n$.

On the other hand, if the initial state is not certain, then the column vector $C_k$ gives the probabilities of the process transitioning (after $n$ transitions) into state $k$ from any one of the possible initial states. Thus a column from the matrix $\mathbf{P}^n$ gives the probability of the process in a given state at the $n$th period from all possible initial states. The following gives the probability distribution for $X_n$ (the state of the Markov chain at time $n$).

$\displaystyle (5) \ \ \ \ \ P[X_n=k]=\sum \limits_{i=0}^\infty P_{i,k}^n \times P[X_0=i]$

The calculation in (5) can be viewed as matrix calculation. If we put the probabilities $P[X_0=i]$ in a row vector, then the product of this row vector with the column vector $C_k$ would be $P[X_n=k]$. The following is (5) in matrix multiplication.

$(6) \ \ \ \ \ P[X_n=k]=\left[\begin{array}{cccccc} P(X_0=0) & P(X_0=1) & \cdots & P(X_0=j) & \cdots & P(X_0=w) \\ \end{array}\right] \left[\begin{array}{c} P_{0k}^n \\ \text{ } \\ P_{1k}^n \\ \vdots \\ P_{jk}^n \\ \vdots \\ P_{wk}^n \end{array}\right]$

Additional Comments on Chapman-Kolmogorov

Even though the matrix calculation for $P_{ij}^n$ should be done using software, it pays to understand the orientation of the matrix $\mathbf{P}^n$. In the preceding section, we learn that a row of $\mathbf{P}^n$ is the conditional distribution $X_n \lvert X_0$ (the state of the process at time $n$ given the initial state). In the preceding section, we also learn that a column in the matrix $\mathbf{P}^n$ gives the probabilities of the process landing in a fixed state $k$ from any one of the initial states.

The Chapman-Kolmogorov equations in (3) tells us that an entry in the matrix $\mathbf{P}^{n+m}$ is simply the product of a row in $\mathbf{P}^n$ and a column in $\mathbf{P}^m$. This observation makes it possible to focus just on the transition probability that is asked in a given problem rather than calculating the entire matrix. For example, to calculate an entry $P_{ij}^2$ of the matrix $\mathbf{P}^2$, multiply a row of $\mathbf{P}$ (the row corresponding to the state $i$) and a column of $\mathbf{P}$ (the row corresponding to state $j$). There is no need to calculate the entire matrix $\mathbf{P}^2$ if the goal is just one entry of $\mathbf{P}^2$.

To calculate an entry of $\mathbf{P}^n$, there is a considerable amount of variation in multiplication approaches. For example, multiple a row of $\mathbf{P}$ and a column of $\mathbf{P}^{n-1}$ or multiple a row of $\mathbf{P}^{n-1}$ and a column of $\mathbf{P}$. Or multiple a row of $\mathbf{P}^2$ and a column of $\mathbf{P}^{n-2}$.

If a row of transition probabilities multiplies a transition matrix, the result is a row in a higher transition probability matrix. For example, if a row in $\mathbf{P}$ multiplies the matrix $\mathbf{P}^n$, the result is a row in $\mathbf{P}^{n+1}$. More specifically, the first row in $\mathbf{P}$ multiplying the matrix $\mathbf{P}$ gives the first row of $\mathbf{P}^2$. The first row of $\mathbf{P}$ multiplying the matrix $\mathbf{P}^2$ gives the first row in $\mathbf{P}^3$, etc.

On the other hand, when a transition probability matrix multiplies a column of transition probabilities, the result is a column in a higher transition probability matrix. For example, the matrix $\mathbf{P}^n$ multiplies a column in $\mathbf{P}$, the result is a column in $\mathbf{P}^{n+1}$. More specifically, when the matrix $\mathbf{P}$ multiples the first column in the matrix $\mathbf{P}$, the result is the first column of $\mathbf{P}^2$. When the matrix $\mathbf{P}^2$ multiples a column in the matrix $\mathbf{P}$, the result is the first column in $\mathbf{P}^3$, etc.

Examples

We now give some examples on how to calculate transition probabilities.

Example 1
Consider a Markov chain with the following transition probability matrix.

$\mathbf{P} = \bordermatrix{ & 0 & 1 & 2 \cr 0 & 0.6 & 0.2 & 0.2 \cr 1 & 0.3 & 0.5 & 0.2 \cr 2 & 0.4 & 0.3 & 0.3 \cr } \qquad$

Determine $P[X_3=1 \lvert X_0=0]$ and $P[X_5=2 \lvert X_2=0]$.

Of course, the most straightforward way would be to calculate $\mathbf{P}^3$. Then $P[X_3=1 \lvert X_0=0]=P_{01}^3$ would be the (0,1)th entry of $\mathbf{P}^3$ and $P[X_5=2 \lvert X_2=0]=P_{02}^3$ would be the (0,1)th entry of $\mathbf{P}^3$. In fact, it is a good practice to use an online matrix calculator for this task. Doing so produces the following marrices.

$\mathbf{P}^2 = \bordermatrix{ & 0 & 1 & 2 \cr 0 & 0.50 & 0.28 & 0.22 \cr 1 & 0.41 & 0.37 & 0.22 \cr 2 & 0.45 & 0.32 & 0.23 \cr } \qquad \ \ \ \ \ \ \mathbf{P}^3 = \bordermatrix{ & 0 & 1 & 2 \cr 0 & 0.472 & 0.306 & 0.222 \cr 1 & 0.445 & 0.333 & 0.222 \cr 2 & 0.458 & 0.319 & 0.223 \cr } \qquad$

From the matrix $\mathbf{P}^3$, we see that $P_{01}^3=0.306$ and $P_{02}^3=0.222$. Note that $P_{02}$ is calculated in the introductory example earlier.

Example 2
For the transition matrix $\mathbf{P}$ in Example 1, use the ideas discussed in the section Additional Comments on Chapman-Kolmogorov to compute the first row and the first column in the transition matrix $\mathbf{P}^4$.

$\left[\begin{array}{ccc} 0.6 & 0.2 & 0.2 \\ \end{array}\right] \left[\begin{array}{ccc} 0.472 & 0.306 & 0.222 \\ 0.445 & 0.333 & 0.222 \\ 0.458 & 0.319 & 0.223 \end{array}\right] = \left[\begin{array}{ccc} 0.4638 & 0.314 & 0.2222 \end{array}\right]= \left[\begin{array}{ccc} P_{00}^4 & P_{01}^4 & P_{02}^4 \end{array}\right]$

$\left[\begin{array}{ccc} 0.472 & 0.306 & 0.222 \\ 0.445 & 0.333 & 0.222 \\ 0.458 & 0.319 & 0.223 \end{array}\right] \left[\begin{array}{c} 0.6 \\ 0.3 \\ 0.4 \end{array}\right] = \left[\begin{array}{c} 0.4638 \\ 0.4557 \\ 0.4597 \end{array}\right]= \left[\begin{array}{c} P_{00}^4 \\ P_{10}^4 \\ P_{20}^4 \end{array}\right]$

As discussed in earlier, the first row of $\mathbf{P}^4$ is obtained by multiplying the first row of $\mathbf{P}$ with the matrix $\mathbf{P}^3$. The first column of $\mathbf{P}^4$ is obtained by multiplying the matrix $\mathbf{P}^3$ with the first column of the matrix $\mathbf{P}$.

Example 3
A particle moves among states 0, 1 and 2 according to a Markov process with the following transition probability matrix.

$\mathbf{P} = \bordermatrix{ & 0 & 1 & 2 \cr 0 & 0.6 & 0.3 & 0.1 \cr 1 & 0.3 & 0.3 & 0.4 \cr 2 & 0.3 & 0.2 & 0.5 \cr } \qquad$

Let $X_n$ be the position of the particle after the $n$th move. Suppose that at the beginning, the particle is in state 1. Determine the probability $P[X_2=k]$ where $k=0,1,2$.

Since the initial state of the process is certain, $P[X_2=k]=P[X_2=k \lvert X_0=1]=P_{1k}^2$. Thus the problem is to find the second row of $\mathbf{P}^2$.

$\left[\begin{array}{ccc} 0.3 & 0.3 & 0.4 \\ \end{array}\right] \left[\begin{array}{ccc} 0.6 & 0.3 & 0.1 \\ 0.3 & 0.3 & 0.4 \\ 0.3 & 0.2 & 0.5 \end{array}\right] = \left[\begin{array}{ccc} 0.39 & 0.26 & 0.35 \end{array}\right]= \left[\begin{array}{ccc} P_{10}^2 & P_{11}^2 & P_{12}^2 \end{array}\right]$

Example 4
Consider a Markov chain with the following transition probability matrix.

$\mathbf{P} = \bordermatrix{ & 0 & 1 & 2 \cr 0 & 0.3 & 0.2 & 0.5 \cr 1 & 0.1 & 0.3 & 0.6 \cr 2 & 0.4 & 0.3 & 0.3 \cr } \qquad$

Suppose that initially the process is equally likely to be in state 0 or state 1. Determine $P[X_3=0]$ and $P[X_3=1]$.

To find $P[X_3=0]$, we need the first column of $\mathbf{P}^3$. To find $P[X_3=1]$, we need the second column of $\mathbf{P}^3$. Using an online calculator produces $\mathbf{P}^3$.

$\mathbf{P}^3 = \bordermatrix{ & 0 & 1 & 2 \cr 0 & 0.288 & 0.269 & 0.443 \cr 1 & 0.283 & 0.270 & 0.447 \cr 2 & 0.295 & 0.273 & 0.432 \cr } \qquad$

The following calculation gives the answers.

$P[X_3=0]=\left[\begin{array}{ccc} 0.5 & 0.5 & 0 \\ \end{array}\right] \left[\begin{array}{c} 0.288 \\ 0.283 \\ 0.295 \end{array}\right]=0.2855$

$P[X_3=1]=\left[\begin{array}{ccc} 0.5 & 0.5 & 0 \\ \end{array}\right] \left[\begin{array}{c} 0.269 \\ 0.270 \\ 0.273 \end{array}\right]=0.2695$

$P[X_3=2]=1-P[X_3=0]-P[X_3=1]=0.445$

Example 5
Consider a Markov chain with the following transition probability matrix.

$\mathbf{P} = \bordermatrix{ & 0 & 1 & 2 \cr 0 & 0.6 & 0.2 & 0.2 \cr 1 & 0.2 & 0.2 & 0.6 \cr 2 & 0 & 0 & 1 \cr } \qquad$

At time 0, the Markov process is in state 0. Let $T=\text{min}\left\{n \ge 0:X_n=2 \right\}$. Note that state 2 is called an absorbing state. The Markov process will eventually reach and be absorbed in state 2 (it stays there forever whenever the process reaches state 2). Thus $T$ is the first time period in which the process reaches state 2. Suppose that the Markov process is being observed and that absorption has not taken place. Then we would be interested in this conditional probability: the probability that the process is in state 0 or state 1 given that the process has not been absorbed. Determine $P[X_4=0 \lvert T>4]$.

The key idea here is that for the event $T>4$ to happen, $X_4=0$ or $X_4=1$. Thus

$P[T>4]=P[X_4=0 \text{ or } X_4=1]=P[X_4=0]+P[X_4=1]=P_{00}^4+P_{01}^4$.

Note that since the initial state is certain to be state 0, $P[X_4=0]=P_{00}^4$ and $P[X_4=1]=P_{01}^4$. Using an online calculator gives the following matrix.

$\mathbf{P}^4 = \bordermatrix{ & 0 & 1 & 2 \cr 0 & 0.1856 & 0.0768 & 0.7376 \cr 1 & 0.0768 & 0.032 & 0.8912 \cr 2 & 0 & 0 & 1 \cr } \qquad$

The following gives the desired result.

$\displaystyle P[X_4=0 \lvert T>4]=\frac{P_{00}^4}{P_{00}^4+P_{01}^4}=\frac{0.1856}{0.1856+0.0768}=0.7073$

Thus if absorption has not taken place in the 4th period, the process is in state 0 about 71% of the time.

Example 6
Continue with the Markov chain in Example 5. Determine $P[T=4]$.

Note that $P[T=4]=P[T>3]-P[T>4]$. The probability $P[T>4]$ is already determined using $\mathbf{P}^4$. To determine $P[T>3]$, we need the top row of $\mathbf{P}^3$.

$\mathbf{P}^3 = \bordermatrix{ & 0 & 1 & 2 \cr 0 & 0.272 & 0.112 & 0.616 \cr 1 & 0.112 & 0.048 & 0.84 \cr 2 & 0 & 0 & 1 \cr } \qquad$

Thus $P[T>3]=P_{00}^3+P_{01}^3=0.272+0.112=0.384$. Thus $P[T=4]=0.384-0.2624=0.1216$. Thus about 12% of the time, absorption takes place in the 4th period about 12% of the time.

Exercises

We give a few practice problems to reinforce the concepts and calculation discussed here. Further practice problems are in a companion blog (here and here).

Exercise 1
Consider a Markov chain with the following transition probability matrix.

$\mathbf{P} = \bordermatrix{ & 0 & 1 & 2 \cr 0 & 0.1 & 0.2 & 0.7 \cr 1 & 0.2 & 0.2 & 0.6 \cr 2 & 0.6 & 0.1 & 0.3 \cr } \qquad$

Determine these conditional probabilities.

• $P[X_3=2 \lvert X_1=1]$
• $P[X_3=2 \lvert X_0=1]$
• $P[X_4=2 \lvert X_0=1]$

Exercise 2
A particle moves among states 1, 2 and 3 according to a Markov process with the following transition probability matrix.

$\mathbf{P} = \bordermatrix{ & 1 & 2 & 3 \cr 1 & 0 & 0.6 & 0.4 \cr 2 & 0.6 & 0 & 0.4 \cr 3 & 0.6 & 0.4 & 0 \cr } \qquad$

Let $X_n$ be the position of the particle after the $n$th move. Determine the probability $P[X_3=1 \lvert X_0=1]$ and $P[X_4=1 \lvert X_0=1]$.

Exercise 3
Consider a Markov chain with the following transition probability matrix.

$\mathbf{P} = \bordermatrix{ & 0 & 1 & 2 \cr 0 & 0.1 & 0.2 & 0.7 \cr 1 & 0.9 & 0.1 & 0 \cr 2 & 0.1 & 0.8 & 0.1 \cr } \qquad$

Suppose that initially the process is equally likely to be in state 0 or state 1 or state 2. Determine the distribution for $X_3$.

Exercise 4
Continue with Example 5 and Example 6. Work these two examples assuming that the Markov chain starts in state 1 initially.

$\text{ }$

$\text{ }$

$\text{ }$

Answers to Exercises

Exercise 1

• $P[X_3=2 \lvert X_1=1]=P_{12}^2=0.13$
• $P[X_3=2 \lvert X_0=1]=P_{12}^3=0.16$
• $P[X_4=2 \lvert X_0=1]=P_{12}^4=0.1473$

Exercise 2

• $P[X_3=1 \lvert X_0=1]=P_{11}^3=0.24$
• $P[X_4=1 \lvert X_0=1]=P_{11}^4=0.456$

Exercise 3

• $\displaystyle P[X_3=0]=\frac{1.076}{3}=0.3587$
• $\displaystyle P[X_3=1]=\frac{1.013}{3}=0.3377$
• $\displaystyle P[X_3=2]=\frac{0.911}{3}=0.3037$

Exercise 4

• $\displaystyle P[X_4=0 \lvert T>4]=\frac{P_{10}^4}{P_{10}^4+P_{11}^4}=\frac{0.0768}{0.0768+0.0.032}=0.70588$
• $P[T=4]=P[T>3]-P[T>4]=0.16-0.1088=0.0512$

$\text{ }$

$\text{ }$

$\text{ }$

$\copyright$ 2017 – Dan Ma