5.2 Information Measurement

The engineer engaged in system design has a constant concern, which is to estimate the resources that must be deployed in order to transport and store information as efficiently and reliably as possible. It is a matter of determining the format or message coding or data block containing the information. It amounts to estimating the quantity of information contained in the message and then to determining the resources to allocate (in terms of bitrate, storage capacity, etc.).

Obviously, the system is designed to work whichever message is sent and whatever information it carries. The distinction should be clearly established between the concepts of information and message: the message carries the information -unknown and thus random from the addressee's standpoint.

5.2.1 Algebraic Definition of Information

Let ? be a finite set of messages, assumed to be equiprobable, and let ? denote the cardinal of this set. Then the quantity of information contained in a message drawn from ? is defined as:

Indeed, the complexity of physical systems grows exponentially with the number of its elements. Obviously for a digital message built with 0s and 1s, the possible number of arrangements doubles for any additional bit, increasing by 1 its base-2 logarithm.

5.2.2 Probabilistic Definition of Information

Here we no longer consider all messages as equiprobable. Noting as P( ? _i) the occurrence probability of message ? _i, we then write:

The similarity with the mathematical form of the entropy...