TABLE OF CONTENTS Information theory provides a quantitative measure of the information contained in message signals and allows us to determine the capacity of a communication system to transfer this information from source to destination. In this chapter we briefly explore some basic ideas involved in the information theory and source coding.
An information source is an object that produces an event, the outcome of which is selected at random according to a probability distribution. A practical source in a communication system is a device that produces messages, and it can be either analog or discrete. In this chapter we deal mainly the discrete sources, since analog sources can be transformed to discrete sources through the use of sampling and quantization techniques, described in Chap. 5. A discrete information source is a source that has only a finite set of symbols as possible outputs. The set of source symbols is called the source alphabet, and the elements of the set are called symbols or letters.
Information sources can be classified as having memory or being memoryless. A source with memory is one for which a current symbol depends on the previous symbols. A memoryless source is one for which a current symbol produced is independent of the previous symbols.
A discrete memoryless source (DMS) can be characterized by the list of the symbols, the probability assignment to these symbols, and the specification of the rate of generating these symbols by the...
