TABLE OF CONTENTS In order to define entropy, let us consider a memoryless source, in its stationary state. That means that the successive symbols it sends are independent, obeying a common time-independent probability distribution.
We consider a memoryless source S, sending symbols s i with probability p i. The entropy H( S) of the source is the average self-information it generates, that is:
More generally, given a discrete random variable X = { x 1, x 2, , x n} on ?, its entropy is defined as:
Clearly, this represents the a priori average uncertainty of the event ( X).
A binary source is an important example. The corresponding random variable has a value 1 with probability p 1 = p and a value 0 with probability p 2 = 1 ? p. Its entropy is thus given by:
Let us consider in more detail the function H 2( p):
for p = 0 and p = 1, then H 2 ( p) = 0. Obviously in these two cases there is no uncertainty on the output;
continuity: the entropy H( X) = H 2( p i) is a continuous function of p i(sum of products of continuous functions here the logarithm);
the function is...
