TABLE OF CONTENTS We now present the basic principles of source coding, from the standpoint of information theory and specifically the notions of code efficiency and redundancy. Here we restrict ourselves to the case of memoryless sources, keeping in mind that the notions of adjoint source and extensions, already presented, allow us to extend these principles to sources with memory, such as Markov sources.
We have seen previously that the source symbols s i are translated into codewords c i. Naturally, the main goal of this coding is to reach the maximum efficiency in the transport of information.
In order to quantify this efficiency, we associate a cost with the message, corresponding to its length. Actually, we can consider that the longer the message, the higher its transmission cost.
Let us denote as l i the length of the codeword c i, i.e. the total number of letters (drawn from an alphabet of l letters) composing the word. We assume that all the letters of the alphabet [ ?] = [ ? 1, ? 2, , ? l] in the codewords [ C] = [ c 1, c 2, , c k] are transmitted at the same speed (e.g., the 0s and 1s are transmitted with the same speed on a digital link). We assume that the codewords and source symbols have the same occurrence probability (there is a direct correspondence...
