When is entropy maximized

To handle varying functions, we will make use of the Calculus of Variations. Sign up to join this community. The best answers are voted up and rise to the top. Stack Overflow for Teams — Collaborate and share knowledge with a private group. Create a free Team What is Teams?

Learn more. Why is Entropy maximised when the probability distribution is uniform? Ask Question. Asked 8 years, 3 months ago. Active 1 year, 7 months ago. Viewed 60k times. Improve this question. How is this statement valid? Isnt the entropy of the uniform distribution the maximum always?

When the constraints are that all probability must vanish beyond predefined limits, the maximum entropy solution is uniform. When instead the constraints are that the expectation and variance must equal predefined values, the ME solution is Gaussian.

The statements you quote must have been made within particular contexts where these constraints were stated or at least implicitly understood. This "differential entropy" is a different animal than the entropy of discrete distributions. The chief difference is that the differential entropy is not invariant under a change of variables. What if there are no constraints? I mean, cant there be a question like this?

Which probability distribution has maximum entropy? Show 2 more comments. Active Oldest Votes. Improve this answer. Show 8 more comments. Community Bot 1. Aksakal Aksakal 53k 5 5 gold badges 84 84 silver badges bronze badges. Add a comment. Octavian Ganea Octavian Ganea 91 1 1 silver badge 1 1 bronze badge.

GZ92 GZ92 31 2 2 bronze badges. Roland Roland 2 2 bronze badges. Mike Hawk Mike Hawk 1 1 silver badge 6 6 bronze badges. I have hopefully covered these issues in my edit. I prefer to avoid the extended reals, so I offer a few other workarounds. Sign up or log in Sign up using Google.

Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. Email Required, but never shown.

Second, the "information" part refers to information theory, which deals with sending messages or symbols over a channel. One crucial point for our explanation is that the "information" of a data source is modelled as a probability distribution.

So everything we talk about is with respect to a probabilistic model of the data. Now let's start from the basic idea of information. Wikipedia has a good article on Shannon's rationale for information, check it out for more details. I'll simplify it a bit to pick out the main points.

First, information was originally defined in the context of sending a message between a transmitter and receiver over a potentially noisy channel. Think about a situation where you are shouting messages to your friend across a large field. You are the transmitter, your friend the receiver, and the channel is this large field.

We can model what your friend is hearing using probability. For simplicity, let's say you are only shouting or transmitting letters of the alphabet A-Z. We'll also assume that the message always transmits clearly if not this will affect your probability distribution by adding noise. Let's take a look at a couple examples to get a feel for how information works:. Suppose you and your friend agree that you will always shout "A" ahead of time or a priori.

So when you actually do start shouting, how much information is being transmitted? None, because your friend knows exactly what you are saying. This is akin to modelling the probability of receiving "A" as 1, and all other letters as 0. Suppose you and your friend agree, a priori, that you will being shouting letters in order from some English text.

Which letter do you think would have more information, "E" or "Z"? Since we know "E" is the most common letter in the English language, we can usually guess when the next character is an "E".

So we'll be less surprised when it happens, thus it has a relatively low amount of information that is being transmitted. Conversely, "Z" is an uncommon letter. So we would probably not guess that it's coming next and be surprised when it does, thus "Z" conveys more information than "E" in this situation.

This is akin to modelling a probability distribution over the alphabet with probabilities proportional to the relative frequencies of letters occurring in the English language. Another way of describing information is a measure of "surprise". If you are more surprised by the result then it has more information. Based on some desired mathematical properties shown in the box below, we can generalize this idea to define information as:.

The base of the logarithm isn't too important since it will just adjust the value by a constant. If something almost always happens e. That is, getting the information for independent events together, or separately, should be the same.

Now that we have an idea about the information of a single event, we can define entropy in the context of a probability distribution over a set of events. Et voila! The usual non-intuitive definition of entropy we all know and love. Entropy, then, is the average amount of information or surprise for an event in a probability distribution. When transmitting English text, the entropy will be the average entropy using letter frequencies 1.

Using Equation 2 using base 2 :. So one bit of information is transmitted with every observation of a fair coin toss. A continuous analogue to discrete entropy is called differential entropy or continuous entropy.

It has a very similar equation using integrals instead of sums:. We have to be careful with differential entropy because some of the properties of discrete entropy do not apply to differential entropy, for example, differential entropy can be negative. The principle of maximum entropy states that given precisely stated prior data, the probability distribution that best represents the current state of knowledge is the one with the largest information entropy.