TABLE OF CONTENTS A Markov chain is a random process with the property that given the values of the process from time zero up through the current time, the conditional probability of the value of the process at any future time depends only on its value at the current time. This is equivalent to saying that the future and the past are conditionally independent given the present (cf. Problem 70 in Chapter 1).
Markov chains often have intuitively pleasing interpretations. Some examples discussed in this chapter are random walks (without barriers and with barriers, which may be reflecting, absorbing, or neither), queuing systems (with finite or infinite buffers), birth death processes (with or without spontaneous generation), life (with states being healthy, sick, and death ), and the gambler s ruin problem.
Section 12.1 briefly highlights some simple properties of conditional probability that are very useful in studying Markov chains. Sections 12.2 12.4 cover basic results about discrete-time Markov chains. Continuous-time chains are discussed in Section 12.5.
[ ] Sections 12.1 12.4 can be covered any time after Chapter 3. However, Section 12.5 uses material from Chapter 5 and Chapter 11.
We present some easily-derived properties of conditional probability. These observations will greatly simplify some of our calculations for Markov chains. 1
Given any event A and any two integer-valued random variables X and Y, show that if P( A X = i, Y = j) depends on i but not j,...
