# Definition:Entropy (Probability Theory)

Jump to navigation
Jump to search

## Definition

Let $X$ be a discrete random variable that takes on the values of $\set {x_1, x_2, \ldots, x_n}$ and has a probability mass function of $\map p {x_i}$.

Then the **entropy** of $X$ is:

- $\ds \map H X := -\sum_{i \mathop = 1}^n \map p {x_i} \log_2 \map p {x_i}$

and is measured in units of **bits**.

By convention $0 \log_2 0 = 0$, which is justified since $\ds \lim_{x \mathop \to 0^+} x \log_2 x = 0$.