Information theory Shannon work Founders mathematics
  

Information Theory was born in 1948 out of Claude Shannon's landmark paper, "A Mathematical Theory of Communication. " As a scientific field it lies somewhere between Communication Theory, Statistics, and Probability Theory.

Shannon's remarkable discoveries immediately grasped the attention of the world mathematical community. One of the most prominent mathematicians of our century, A. N. Kolmogorov, generalized Shannon's ideas, and put them on a slightly more rigorous mathematical basis. These ideas had a great influence in a wide range of mathematical fields, among which the new field of the entropy theory of abstract dynamical systems.

One of the most prominent features of 20th-century technology has been the development and exploitation of new communications media. Concurrent with the growth of devices for transmitting and processing information, a unifying theory known as information theory was developed and became the subject of intensive research. It is an example of a theory that was initiated primarily by one man, the U.S. electrical engineer, Claude E. Shannon, whose initial ideas appeared in the article "The Mathematical Theory of Communication" in the Bell System Technical Journal (1948). In its broadest sense, information is interpreted to include the messages occurring in any of the standard communications media, such as telegraphy, radio, or television, and the signals involved in electronic computers, servomechanism systems, and other data-processing devices. The theory is even applied to the signals appearing in the nerve networks of humans and other animals. The signals or messages do not have to be meaningful in any ordinary sense.

The chief concern of information theory is to discover mathematical laws governing systems designed to communicate or manipulate information. It sets up quantitative measures of information and of the capacity of various systems to transmit, store, and otherwise process information.

Some of the problems treated are related to finding the best methods of using various available communication systems and the best methods for separating the wanted information, or signal, from the extraneous information, or noise. Another problem is the setting of upper bounds on what it is possible to achieve with a given information-carrying medium (often called an information channel). While the central results are chiefly of interest to communication engineers, some of the concepts have been adopted and found useful in such fields as psychology and linguistics. The boundaries of information theory are quite vague. The theory overlaps heavily with communication theory but is more oriented toward the fundamental limitations on the processing and communication of information and less oriented toward the detailed operation of the devices employed.

I. Information Theory:

One of the basic postulates of information theory is that information can be treated like a measurable physical quantity, such as density or mass. The theory has widely applied by communication engineers and some of its concepts have found application in psychology and linguistics.

The basic elements of any general communications system include
  1. a source of information which is a transmitting device that transforms the information or "message" into a form suitable for transmission by a particular means.
  2. the means or channel over which the message is transmitted.
  3. a receiving device which decodes the message back into some approximation of its original form.
  4. the destination or intended recipient of the message.
  5. a source of noise (i.e., interference or distortion) which changes the message in unpredictable ways during transmission.
It is important to note that "information" as understood in information theory has nothing to do with any inherent meaning in a message.
It is rather a degree of order, or nonrandomness, that can be measured and treated mathematically much as mass or energy or other physical quantities are. A mathematical characterization of the generalized communication system yields a number of important quantities, including
  1. the rate at which information is produced at the source.
  2. the capacity of the channel for handling information.
  3. the average amount of information in a message of any particular type.
To a large extent the techniques used with information theory are drawn from the mathematical science of probability. Estimates of the accuracy of a given transmission of information under known conditions of noise interference, for example, are probabilistic, as are the numerous approaches to encoding and decoding that have been developed to reduce uncertainty or error to minimal levels.

Information and Uncertainty are technical terms that describe any process that selects one or more objects from a set of objects. We won't be dealing with the meaning or implications of the information since nobody knows how to do that mathematically.

Suppose we have a device that can produce 3 symbols, A, B, or C. As we wait for the next symbol, we are uncertain as to which symbol it will produce. Once a symbol appears and we see it, our uncertainty decreases, and we remark that we have received some information. That is, information is a decrease in uncertainty.

How should uncertainty be measured? The simplest way should be to say that we have an "uncertainty of 3 symbols". This would work well until we begin to watch a second device at the same time, which, let us imagine, produces symbols 1 and 2. The second device gives us an "uncertainty of 2 symbols". If we combine the devices into one device, there are six possibilities, A1, A2, B1, B2, C1, C2. This device has an "uncertainty of 6 symbols". This is not the way we usually think about information, for if we receive two books, we would prefer to say that we received twice as much information than from one book. That is, we would like our measure to be additive.

II. Symbolic Logic and Switching Theory:

Shannon is as the founding father of electronic communications age since he noticed and discovered the similarity between Boolean algebra and the telephone switching circuits. For example, the fundamental unit of information is a yes-no situation. Either something is or is not. This can be easily expressed in Boolean two-value binary algebra by 1 and 0, so that 1 means "on" when the switch is closed and the power is on, and 0 means "off" when the switch is open and power is off.


Under these circumstances, 1 and 0 are binary digits, a phrase that can be shortened to "bits." Thus the unit of information is the bit. A more complicated information can be viewed as built up out of combinations of bits.

By 1948, He turned his efforts toward a fundamental understanding of the problem and had evolved a method of expressing information in quantitative form.