Conditional entropy - Definition and Overview

The conditional entropy is an entropy measure used in information theory. The conditional entropy measures how much entropy a random variable <math>Y<math> has remaining if we have already learned completely the value of a second random variable <math>X<math>. It is referred to as the entropy of <math>Y<math> conditional on <math>X<math>, and is written <math>H(Y|X)<math>. Like other entropies, the conditional entropy is measured in bits.

Given random variables <math>X<math> and <math>Y<math> with entropies <math>H(X)<math> and <math>H(Y)<math>, and with a joint entropy <math>H(X,Y)<math>, the conditional entropy of <math>Y<math> given <math>X<math> is defined as <math>H(Y|X) \equiv H(X,Y) - H(X)<math>. Intuitively, the combined system contains <math>H(X,Y)<math> bits of information. If we learn the value of <math>X<math>, we have gained <math>H(X)<math> bits of information, and the system has <math>H(Y|X)<math> bits remaining.

<math>H(Y|X)=0<math> if and only if the value of <math>Y<math> is completely determined by the value of <math>X<math>. Conversely, <math>H(Y|X) = H(Y)<math> if and only if <math>Y<math> and <math>X<math> are independent random variables.

In quantum information theory, the conditional entropy is generalized to the conditional quantum entropy.