Talk:Gibbs' inequality

From Wikipedia, the free encyclopedia

The proof on this page seems to state that -\sum_{i\in I} p_i\,\textrm{ln}\,p_i = -\sum_{i=1}^n p_i\, \textrm{ln} \,p_i. However, I was defined so that pi = 0 whenever i\notin I, so making this change in the indices for the sum introduces indeterminate terms of the form 0\cdot \textrm{ln}\, 0 = 0 \cdot -\infty.

Changing the indices for the sum may introduce similar indeterminate forms on the left hand side of that inequality, as well.

How can this proof be corrected?


The notation used on this page is lazy. The implicit understanding is that a new function is defined as:
\ln^{+}(x) = \begin{cases} \ln(x) & \mbox{ if } x > 0 \\ 0 & \mbox{ if } x <= 0 \end{cases}
but the author (me - reetep) didn't bother to specify this explicitly. My apologies - it's the sort of thing you do without thinking after having completed a mathematics degree; you become accustomed to writing mathematics for people with a similar background in maths and more often than not you find yourself demonstrating how something might be proved without actually doing it (because you have neglected to complete all of the technical details).

If you refer to the definition of mathematical entropy you should find there that a similar definition has been made for the log function. This is essentially what Gibbs' inequality is about and we are talking about the same function in this article.

[edit] Special case

Consider a two-state distribution with probabilities p_1=0.5,\ p_2=0.5 and alternative probabilities q_1=0,\ q_2=1. This is allowed.

The inequality will then look like this -(p_1 ln(q_1) + p_2 ln(q_2))\geq -(p_1 ln(p_1) + p_2 ln(p_2)). This results in -0 \geq ln(2), which is wrong.

What constrains did I not obey?

134.100.209.149 (talk) 16:01, 3 January 2008 (UTC)

[edit] Question about format

I'm familiar with the traditional way to write entropies in Shannon-Jaynes format, which may a good reason to not fiddle, but a minus sign multiplying the side of an inequality sets off flags in my brain about the danger of reversals if anything gets multiplied by anything. Of course this danger doesn't exist if everything is positive, which at least in the first statement of the inequality could be the case. I've added a comment to that effect which I think solves the problem, but am wondering this: Are there any objections to also writing for example -Σpiln[qi] as Σpiln[1/qi] where it might keep the question from coming up? Thermochap (talk) 19:27, 21 February 2008 (UTC)