TABLE OF CONTENTS Consider a stochastic process { X n; n = 0, 1, } defined on a probability space 
, taking values in a countable set or state space S, which we will assume to be a subset of the nonnegative integers {0, 1, } unless explicitly defined otherwise. If X n ? X n( ?) = i we say the process is in state i at time n.
We say the proccss X n is a homogeneous Markov chain defined on 
, if for all n ? 0 and for any state j and any sequence of preceding states { i 0, i 1, , i n ?1, i}, we have
K is called the probability transition kernel of the Markov chain. We say the initial distribution is ? if P( X 0 = i) = ?( i) and we sometimes denote P by P ?. If ?( i 0) = 1 then denote P by P i 0. Similarly, if the initial distribution is ? or if ?( i 0) = 1 we denote the associated expectations by E ? or E i 0 respectively.
Intuitively the above conditions mean that knowing the...
