User:Comfortably Paranoid/IT

From Wikipedia, the free encyclopedia

[edit] Information Theory Foundations

This page includes results which are used in proving theorems in Information theory.

Sanov's theorem does not have a page. See Cover and Thomas, pg. 292. Young's inequality mentions Sanov's thm.

[edit] Jensen's Inequality

f( \mathbb E [x] ) \leq \mathbb E[ f(x) ]

[edit] Divergence

The Divergence is:

D_{\mathrm{KL}}(P\|Q) = \sum_i P(i) \log \frac{P(i)}{Q(i)} \!