+ PRML 1.31: Consider two variables \(x\) and \(y\) having joint distribution \(p(x,y)\). Show that the differential entropy of this pair of variables satisfies \(H(x,y) \le H(x) + H(y)\) with equality if, and only if, \(x\) and \(y\) are statistically independent. +
+ If \(p(x)\) and \(p(y)\) are independent then the joint distribution is given by:
+ \(p(x,y) = p(x)p(y)\)
+
+ Based on the independent \(p(x)\) and \(p(y)\) the joint entropy can be derived from the conditional entropies \(H(x|y)\) and \(H(y|x)\):
+ \(H(x|y) = H(x)\)
+ \(H(y|x) = H(y) \to\)
+ \(H(x,y) = H(x) + H(y|x) = H(y) + H(x|y) \to\)
+ \(H(x,y) = H(x) + H(y)\)
+
+ Therefore, there is no mutual information \(I(x,y)\) if \(p(x)\) and \(p(y)\) are independent:
+ \(H(x,y) = H(x) + H(y)\ \to\)
+ \(I(x,y) = H(x) + H(y) - H(x,y) = H(x,y) - H(x,y) = 0\)
+
+ If \(p(x)\) and \(p(y)\) are dependent:
+ \(H(x,y) < H(x) + H(y)\ \to\)
+ \(I(x,y) = H(x) + H(y) - H(x,y) > 0\)
+
+ The indepent and dependent case can be combined to a general form:
+ \(H(x,y) \le H(x) + H(y)\ \to\)
+ \(I(x,y) = H(x) + H(y) - H(x,y) \ge 0\)
+
Start here. +
+Start here. +
+ The Kullback–Leibler divergence \(D_{KL}(P||Q)\) between \(P\) and \(Q\) is defined as:
+ \(D_{KL}(P||Q) = \sum_{x}P(x) \log_2\left(\frac{P(x)}{Q(x)}\right)\)
+
+ The lower bound is reached when \(P(x) = Q(x)\) because \(\left(\frac{P(x)}{Q(x)}\right) = 1\):
+ \(D_{KL}(P||Q) = \sum_{x}P(x) \log_2(1) = \sum_{x}P(x) 0 = 0\)
+
+ The upper bound is reached when \(Q(x)\) is disjoint from \(P(x)\), i.e., \(Q(x)\) is zero where \(P(x)\) is not zero, because then the log-ratio \(\log_2\left(\frac{P(x)}{Q(x)}\right)\) becomes \(\infty\):
+ \(x_i \in x\)
+ \(P(x_i) \log_2\left(\frac{P(x_i)}{Q(x_i)}\right) = P(x_i) \log_2\left(\frac{P(x_i)}{0}\right) = \infty \to\)
+ \(D_{KL}(P||Q) = \sum_{x}P(x) \log_2\left(\frac{P(x)}{Q(x)}\right) = \infty\)
+