Dirichlet Process for dummies
A promise is a promise, here is a post on the so-called Dirichlet process, or DP.
What is it? a stochastic process, whose realizations are probability measures. So it is a distribution on distributions. A nice feature for nonparametric Bayesians, who can thus use the DP as a prior when the parameter itself is a probability measure.
As mentionned in an earlier post, a foundational paper and still a nice reading today, which introduced the DP, is Ferguson, T.S. (1973), A Bayesian Analysis of Some Nonparametric Problems. The Annals of Statistics, 1, 209-230. I will not go in very deep details here, but mainly will stress the discreteness of the DP.
First, a DP, say on , has two parameters, a precision parameter , and a base measure on . Basically, is the mean of the process, and measures the inverse of its variance. Formally, we write for a value of the DP. Then, for all measurable subset of , , and . Actually, a more acurate statement says that .
A realization is almost surely discrete. In other words, it is a mixture of Dirac masses. Let us explain this explicit expression as a countable mixture, due to Sethuraman (1994). Let , and , mutually independent. Define , and . Then writes . This is called the Sethuraman representation, also refered to as “stick-breaking”. The reason for the name is in the definition of the weights : each can be seen as the length of a part of a stick of unit lenght, broken in infinitely many sticks. The first stick is of length . The remaining part has length , and is broken at of its length, which defines a second stick of length . And so forth. We see easily that this builds a sequence of s that sum to 1, because the remaining part at step has length , which goes to 0 almost surely.
Now let us illustrate this with the nice plots of Eric Barrat. He chooses a standard normal for , which is quite usual, and . A way to get a graphical view of a realization is to represent a Dirac mass by its weight: