Conditional Entropy Decreases if More Given

Theorem
Let $\struct {\Omega, \Sigma, \Pr}$ be a probability space.

Let $\AA, \CC, \DD \subseteq \Sigma$ be finite sub-$\sigma$-algebras.

Then:
 * $\CC \subseteq \DD \implies \map H {\AA \mid \CC} \ge \map H {\AA \mid \DD}$

where:
 * $\map H {\cdot \mid \cdot}$ denotes the conditional entropy

Proof
Consider the generated finite partitions:
 * $\xi := \map \xi \AA$
 * $\eta := \map \xi \CC$
 * $\gamma := \map \xi \DD$

By Generating Partition Preserves Order, $\CC \subseteq \DD$ implies:
 * $\eta \le \gamma$

where $\le$ denotes the refinement.

By, we shall show:
 * $\map H {\xi \mid \eta} \ge \map H {\xi \mid \gamma}$

Recall that $\eta \le \gamma$ implies:
 * $\forall B \in \eta, \forall D \in \gamma \implies D \subseteq B \; \text{or} \; B \cap D = \O$

In particular, we have:


 * $(1): \quad \forall B \in \eta, \forall D \in \gamma: \map \Pr {B \cap D} = \begin{cases}

\map \Pr D &: D \subseteq B \\ 0 &: B \cap D = \O \end{cases}$

Thus, for all $A \in \xi$ and $B \in \gamma$ such that $\map \Pr B > 0$:

The function $\phi$ used in is concave since the second derivative is negative:
 * $\forall x > 0 : \map {\phi ' '} x = -\dfrac 1 x < 0$

Note that the concavity holds including $x = 0$, since $\phi$ is continuous there.

So we obtain:

Therefore: