The previous post introduces the notion of Markov chains, more specifically discrete Markov chains. This and the next post focus on calculating transition probabilities.
Suppose that is a Markov chain with transition probability matrix . Elements of are the one-step transition probabilities . As the Markov process moves through the states over time, the probabilities in the matrix shows how likely the process will transition to state in the next time period if the process is currently in state . Elements of are conditional probabilities:
The transition probabilities described in (1) are independent of the time period . These are called stationary transition probabilities. Markov chains with stationary transition probabilities are called stationary Markov chains or time-homogeneous Markov chains. One immediate goal is to develop the -step transition probabilities from the one-step transition probabilities . This is to be developed in the next post. This post focuses on basic calculations with .
Examples of Conditional Probabilities
We use examples to demonstrate conditional probabilities that can be calculated from the one-step transition probabilities . The calculation is based on the following transition probability matrix.
Determine the following conditional probabilities:
Note that the first two are identical since the transition probabilities are stationary. To calculate the first and third probabilities, it is a matter of linking up the one-step transition probabilities from one state to the next.
Example 1 shows that the probability of a chain moving from along a path conditional on a starting state is simply the product of the transition probabilities on that path. We now calculate unconditional probability of a path. For this to be possible, we need to assume a distribution for the initial state.
Assume that , and . Determine the following unconditional probabilities:
To calculate these unconditional probabilities, we multiply the transition probabilities on the path along with the probability of the first state in the path.
To calculate the second probability, we need to first compute . This can be done by conditioning on the initial state .
The following gives the third probability.
Note that each unconditional probability in this example is obtained by multiplying the corresponding conditional probability in Example 1 with the probability of the conditioned event.
Sometimes we know for certain that the Markov chain starts in a given state. Then the unconditional probability of a path is simply the conditional probability of that path (conditioned on the initial state). Determine the three probabilities in Example 2 given that the chain starts in state 0 in the first two probabilities and the chain starts in state 1 in the third probability. The probabilities are:
For the first probability, . With , , the answer from Example 1.
For the second probability, we need to first calculate . Since , . Thus , which is 0.6 times the corresponding conditional probability in Example 1.
Similarly, the third probability is
Conditional Probabilities of Paths in Markov Chains
The conditional probabilities in the above examples are not difficult to do as long as the basic idea of conditional probabilities and the Markov property are understood. First, the idea of conditional probability. The idea is that the probability of the intersection of two events can be obtained by conditioning on one of the events. For example, given two events and , we have
The probability of the intersection of two events is the product of a conditional probability of one event given another event times the probability of the event being conditioned. Relation (2b) is the probability of the event given that event has occurred. Of course, the conditioned event has to have positive probability, i.e. .
Suppose that the event is , the discrete random variable taking on the value and that the event is . Then we have the following relations.
We can also write the conditional probability of conditioning on the event . Suppose we have a path in a Markov chain, . This is the event that the chain starts in state initially, then move to state in time 1 and so on and at time the chain is in state . Conditioning on the initial state, consider the following conditional probability.
Relation (4) can be further simplified since the random variables satisfy the Markov property.
Relation (5) says that the probability of a path in a Markov chain conditioning on the initial state is simply the product of the transition probabilities along the path. This is derived using the Markov property. Relation (5) can be established by an induction argument on the subscript , i.e. the length of a path.
Clearly . This is the fundamental assumption in Markov chains. Now consider .
Note that in the above derivation. This is Markov property in action since the probability of the future state depends only on the state immediately preceding it. Now that relation (5) is true for a path of length 2, the following shows how it is derived for length 3.
Note that going from the third line to the fourth line the Markov property is used. Relation (5) for length 2 is used at the end. The inductive proof keeps going and relation (5) is true for any path of arbitrary length.
The conditional probability in (4) by itself may not be special. With the additional assumption of Markov property, the conditional probability can be made to be a “chain” of one-step transition probabilities.
Using the idea of (3a), the unconditional probability of a path is:
Relation (6) says that the unconditional probability of a path in a Markov chain is simply the conditional probability of the path conditioning on the initial state times the probability of the initial state.
The earlier examples demonstrate relation (5) and relation (6). We now work a few more examples.
There is a slightly different way to look at one unconditional probability in Example 2. The first state in this path is . We can condition on the initial state .
Then every one of the unconditional probability within the summation sign can be computed according to the idea in (6). Note that Example 2 gives the distribution of the initial state.
In calculating probabilities in a Markov chain, conditioning on the first step is a useful technique that will be used often in the subsequent posts.
Consider a Markov chain with the following transition probability matrix.
State 3 is called an absorbing state since . In other words, when the chain reaches state 3, it will never leave. Suppose that initially the Markov chain is in state 0 50% of the time, in state 1 30% of the time and in state 2 20% of the time. Calculate the probability .
We can condition on the initial state as in Example 4.
This is Example 1 in this previous post.
When a binary digit, 0 or 1, is transmitted through a communication system, it passes through several stages. At each stage, there is a probability that the digit is transmitted in error. Let be the digit that is the output at the th stage. Then is a two-state Markov chain with the following transition probability matrix.
Determine the following:
- If the digit to be sent is 0, determine the probability that no error occurs up to the second stage.
- If the digit to be sent is 0, determine the probability that a correct message is received at stage 2 through stage 4.
The first probability is . This is . To calculate the second probability, we can condition on the state in time 1. We first calculate and . Since we know for certain that the initial state is 0, and .
The calculation in this post involves very basic calculation with one-step transition probabilities, i.e. the elements of the transition probability matrix given in a Markov chain. The -step transition probabilities are key to the development of the theory of Markov chains. The next post begins the discussion of Chapman-Kolmogorov equations.
Practice problems are in this companion blog.