Definition:Conditional Entropy of Finite Partitions

Definition

Let $\struct {\Omega, \Sigma, \Pr}$ be a probability space.

Let $\xi, \eta$ be finite partitions of $\Omega$.

The (conditional) entropy of $\xi$ given $\eta$ is defined as:

$\ds \map H {\xi \mid \eta} := \sum_{\substack {B \mathop \in \eta \\ \map \Pr B \mathop > 0}} \map \Pr B \sum_{A \mathop \in \xi} \map \phi {\dfrac {\map \Pr {A \cap B} } {\map \Pr B} }$

where $\phi : \closedint 0 1 \to \R _{\ge 0}$ is defined by:

$\map \phi x := \begin {cases}

0 & : x = 0 \\ -x \map \ln x & : x \in \hointl 0 1 \end {cases}$

Here $\ln$ denotes the natural logarithm.

Also see

• Definition:Entropy of Finite Partition
• Definition:Conditional Entropy of Finite Sub-Sigma-Algebra

Sources

• 2013: Peter Walters: An Introduction to Ergodic Theory (4th ed.) $4.2$: Entropy of a Partition