Hi all,
Last week I attended a wonderful meeting on Approximate Bayesian Computation in Banff, which gathered a nice crowd of ABC users and enthusiasts, including lots of people outside of computational stats, whom I wouldn’t have met otherwise. Christian blogged about it there. My talk on Inference with Wasserstein distances is available as a video here (joint work with Espen Bernton, Mathieu Gerber and Christian Robert, the paper is here). In this post, I’ll summarize a few (personal) points and questions on ABC methods, after recalling the basics of ABC (ahem).
The goal is to learn parameters from a generative model. We know how to sample “fake” data from the model (also called “simulator”, “generator” or “black-box”), given the parameters . We have a prior , and data , where each is -dimensional. We cannot evaluate the likelihood function , thus we cannot apply the usual MLE and Bayesian toolbox. So what can we do?
We can sample parameters from the prior, and sample fake data given these parameters. Some of these fake data will resemble the actual data, in which case we might be interested in the corresponding parameters. More formally, we can sample and until , where is a distance or pseudo-distance between samples (e.g. the Euclidean distance between summary statistics of the samples), and is a threshold. This procedure corresponds to an ABC “rejection sampler”, which targets the so-called ABC posterior distribution, which, itself, approximates a certain distribution as . Essentially, if the discrepancy measure is sensible, and is small enough value, there is hope that the ABC posterior is useful for estimating parameters. Lots of variations of this idea exist: see the bibliography here. Now, some points gathered from the meeting.
Hi! It’s been too long!
In a recent arXiv entry, Espen Bernton, Mathieu Gerber and Christian P. Robert and I explore the use of the Wasserstein distance to perform parameter inference in generative models. A by-product is an ABC-type approach that bypasses the choice of summary statistics. Instead, one chooses a metric on the observation space. Our work fits in the minimum distance estimation framework and is particularly related to “On minimum Kantorovich distance estimators”, by Bassetti, Bodini and Regazzini. A recent and very related paper is “Wasserstein training of restricted Boltzmann machines“, by Montavon, Müller and Cuturi, who have similar objectives but are not considering purely generative models. Similarly to that paper, we make heavy use of recent breakthroughs in numerical methods to approximate Wasserstein distances, breakthroughs which were not available to Bassetti, Bodini and Regazzini in 2006.
Here I’ll describe the main ideas in a simple setting. If you’re excited about ABC, asymptotic properties of minimum Wasserstein estimators, Hilbert space-filling curves, delay reconstructions and Takens’ theorem, or SMC samplers with r-hit kernels, check our paper!
We have data in and a model. A model is a collection of probability distributions on , with d-dimensional parameters to estimate. Problem: you can only simulate from , and not evaluate its probability density function. This is the “ABC” setting of “purely generative” models.
A first step is to view the observations as an empirical distribution , and not as a vector in . It is a very sensible idea for independent data, for which the order should not matter. The paper discusses extensions to dependent data in details.
The next step is inspired by minimum distance estimation principles: we can estimate parameters by minimizing a distance between and , over all . Which distance should we use? In the purely generative case, we can approximate by drawing from it. We are then faced with the problem of computing a distance between two empirical distributions. Many metrics could be envisioned, but the Wasserstein distance is an appealing choice for multiple reasons.
Once the distance is chosen, multiple estimation frameworks are possible. The minimum distance “point” estimator leads to an optimization program, whereas an ABC approach leads to a quasi-posterior distribution to sample, which we tackle with SMC samplers and r-hit kernels.
The theoretical study of the minimum distance estimators associated with Hilbert space-filling curve distances has just started; however we have a variety of results for the standard minimum Wasserstein distance estimator, and for its Bayesian counterpart (see the supplementary materials), including in the misspecified setting. Numerical experiments show very promising performance in a variety of examples, and in particular, we show how careful choices of summary statistics can be completely bypassed. Some examples will be described in future blog entries.
Alan thus decided to re-implement my method and several others (including Christian Robert’s accept-reject algorithm proposed in this paper) in C; see here:
https://github.com/alanrogers/dtnorm
Alan also sent me this interesting plot that compares the different methods. The color of a dot at position (a,b) corresponds to the fastest method for simulating N(0,1) truncated to [a,b];
A few personal remarks:
The Italian mathematician Francesco Faà di Bruno was born in Alessandria (Piedmont, Italy) in 1825 and died in Turin in 1888. At the time of his birth, Piedmont used to be part of the Kingdom of Sardinia, led by the Dukes of Savoy. Italy was then unified in 1861, and the Kingdom of Sardinia became the Kingdom of Italy, of which Turin was declared the first capital. At that time, Piedmontese used to commonly speak both Italian and French.
Faà di Bruno is probably best known today for the eponymous formula which generalizes the derivative of a composition of two functions, , to any order:
over -tuples satisfying
Faà di Bruno published his formula in two notes:
Faà Di Bruno, F. (1857). Note sur une nouvelle formule de calcul différentiel. Quarterly Journal of Pure and Applied Mathematics, 1:359–360. Google Books link.
They both date from December 1855, and were signed in Paris. They are similar and essentially state the formula without a proof. I have arXived a note which contains a translation from the French version to English (reproduced below), as well as the two original notes in French and in Italian. I’ve used for this the Erasmus MMXVI font, thanks Xian for sharing!
NOTE ON A NEW FORMULA FOR DIFFERENTIAL CALCULUS.
By M. Faà di Bruno.
Having observed, when dealing with series development of functions, that there did not exist to date any proper formula dedicated to readily calculate the derivative of any order of a function of function without resorting to computing all preceding derivatives, I thought that it would be very useful to look for it. The formula which I found is well simple; and I hope it shall become of general use henceforth.
Let be any function of the variable , linked to another one by the equation of the form
Denote by the product and by etc the successive derivatives of the function ; the value of will have the following expression:
the sign runs over all integer and non negative values of , for which
the value of being
The expression can also take the form of a determinant, and one has
It is implicit that the exponents of will be considered as orders of derivation.
Paris, December 17. 1855.
Hi,
interested in doing a post-doc with me on anything related to Bayesian Computation? Please let me know, as there is currently a call for post-doc grants at the ENSAE, see below.
Nicolas Chopin
The Labex ECODEC is a research consortium in Economics and Decision Sciences common to three leading French higher education institutions based in the larger Paris area: École polytechnique, ENSAE and HEC Paris. The Labex Ecodec offers:
Two-year postdoctoral fellowships for 2017-2019
The monthly gross salary of postdoctoral fellowships is 3 000 €.
Candidates are invited to contact as soon as possible members of the research group (see below) with whom they intend to work.
Research groups concerned by the call:
Area 1: Secure Careers in a Global Economy
Area 2: Financial Market Failures and Regulation
Area 3: Product Market Regulation and Consumer Decision-Making
Area 4: Evaluating the Impact of Public Policies and Firms’ Decisions
Area 5: New Challenges for New Data
Details of axis can be found on the website:
Deadlines for application:
31^{st} December 2016
Screening of applications and decisions can be made earlier for srong candidates who need an early decision.
The application should be sent to application@labex-ecodec.fr in PDF. Please mention the area number on which you apply in the subject.
The application package includes:
Please note that HEC, Genes, and X PhD students are not eligible to apply for this call.
Selection will be based on excellence and a research project matching the group’s research agenda.
Area 1 “Secure careers in a Global Economy”: Pierre Cahuc (ENSAE), Dominique Rouziès (HEC), Isabelle Méjean (École polytechnique)
Area 2: “Financial Market Failures and Regulation”: François Derrien (HEC), Jean-David Fermanian, (ENSAE) Edouard Challe (École polytechnique)
Area 3: “Decision-Making and Market Regulation”: Nicolas Vieille (HEC), Philippe Choné (ENSAE), Marie-Laure Allain (École polytechnique)
Area 4: “Evaluating the Impact of Public Policies and Firm’s Decisions”: Bruno Crépon (ENSAE), Yukio Koriyama (École polytechnique), Daniel Halbheer (HEC)
Area 5: “New Challenges for New Data”: Anna Simoni (ENSAE), Gilles Stoltz (HEC)
I recently started to review papers on Mathematical Reviews / MathSciNet a decided I would post the reviews here from time to time. Here are the first three which deal with (i) objective Bayes priors for discrete parameters, (ii) random probability measures and inference on species variety and (iii) Bayesian nonparametric asymptotic theory and contraction rates.
The paper deals with objective prior derivation in the discrete parameter setting. Previous treatment of this problem includes J. O. Berger, J.-M. Bernardo and D. Sun [J. Amer. Statist. Assoc. 107 (2012), no. 498, 636–648; MR2980073] who rely on embedding the discrete parameter into a continuous parameter space and then applying reference methodology (J.-M. Bernardo [J. Roy. Statist. Soc. Ser. B 41 (1979), no. 2, 113–147; MR0547240]). The main contribution here is to propose an all purpose objective prior based on the Kullback–Leibler (KL) divergence. More specifically, the prior at any parameter value is obtained as follows: (i) compute the minimum KL divergence over between models indexed by and ; (ii) set proportional to a sound transform of the minimum obtained in (i). A good property of the proposed approach is that it is not problem specific. This objective prior is derived in five models (including binomial and hypergeometric) and is compared to the priors known in the literature. The discussion suggests possible extension to the continuous parameter setting.
A. Lijoi, R. H. Mena and I. Prünster [Biometrika 94 (2007), no. 4, 769–786; MR2416792] recently introduced a Bayesian nonparametric methodology for estimating the species variety featured by an additional unobserved sample of size given an initial observed sample. This methodology was further investigated by S. Favaro, Lijoi and Prünster [Biometrics 68 (2012), no. 4, 1188–1196; MR3040025; Ann. Appl. Probab. 23 (2013), no. 5, 1721–1754; MR3114915]. Although it led to explicit posterior distributions under the general framework of Gibbs-type priors [A. V. Gnedin and J. W. Pitman (2005), Teor. Predst. Din. Sist. Komb. i Algoritm. Metody. 12, 83–102, 244–245;MR2160320], there are situations of practical interest where is required to be very large and the computational burden for evaluating these posterior distributions makes impossible their concrete implementation. This paper presents a solution to this problem for a large class of Gibbs-type priors which encompasses the two parameter Poisson-Dirichlet prior and, among others, the normalized generalized Gamma prior. The solution relies on the study of the large asymptotic behaviour of the posterior distribution of the number of new species in the additional sample. In particular a simple characterization of the limiting posterior distribution is introduced in terms of a scale mixture with respect to a suitable latent random variable; this characterization, combined with the adaptive rejection sampling, leads to derive a large approximation of any feature of interest from the exact posterior distribution. The results are implemented through a simulation study and the analysis of a dataset in linguistics.
A novel prior distribution is proposed for adaptive Bayesian estimation, meaning that the associated posterior distribution contracts to the truth with the exact optimal rate and at the same time is adaptive regardless of the unknown smoothness. The prior is termed \textit{block prior} and is defined on the Fourier coefficients of a curve by independently assigning 0-mean Gaussian distributions on blocks of coefficients indexed by some , with covariance matrix proportional to the identity matrix; the proportional coefficient is itself assigned a prior distribution . Under conditions on , it is shown that (i) the prior puts sufficient prior mass near the true signal and (ii) automatically concentrates on its effective dimension. The main result of the paper is a rate-optimal posterior contraction theorem obtained in a general framework for a modified version of a block prior. Compared to the closely related block spike and slab prior proposed by M. Hoffmann, J. Rousseau and J. Schmidt-Hieber [Ann. Statist. 43 (2015), no. 5, 2259–2295; MR3396985] which only holds for the white noise model, the present result can be applied in a wide range of models. This is illustrated through applications to five mainstream models: density estimation, white noise model, Gaussian sequence model, Gaussian regression and spectral density estimation. The results hold under Sobolev smoothness and their extension to more flexible Besov smoothness is discussed. The paper also provides a discussion on the absence of an extra log term in the posterior contraction rates (thus achieving the exact minimax rate) with a comparison to other priors commonly used in the literature. These include rescaled Gaussian processes [A. W. van der Vaart and H. van Zanten, Electron. J. Stat. 1 (2007), 433–448; MR2357712; Ann. Statist. 37 (2009), no. 5B, 2655–2675; MR2541442] and sieve priors [V. Rivoirard and J. Rousseau, Bayesian Anal. 7 (2012), no. 2, 311–333; MR2934953; J. Arbel, G. Gayraud and J. Rousseau, Scand. J. Stat. 40 (2013), no. 3, 549–570; MR3091697].
I have spent three years as a postdoc at the Collegio Carlo Alberto. This was a great time during which I have been able to interact with top colleagues and to prepare my applications in optimal conditions. Now that I have left for Inria Grenoble, here is a brief picture presentation of the Collegio.
Today, the Collegio is a research center specialized in Economics, Statistics and Social Sciences in general. The Stat team focusses mainly on Bayesian Nonparametrics (Antonio Canale, Pierpaolo De Blasi, Stefano Favaro, Guillaume Kon Kam King, Antonio Lijoi, Bernardo Nipoti, Igor Prünster, Matteo Ruggiero), algebraic statistics (Giovanni Pistone) and functional analysis (Bertrand Lods). An “Allievi Honor Program” is training some of the best University students in order to lead them to US PhD programs.
The Collegio started to train the Turin “elite” several centuries ago. The legend says that the hidden child of Mussolini was sent to school there. Each year, the head of the class had the honor of hanging his portrait (that was a boy school, by the way) in the corridors of the Collegio. Thus achieving posterity! But the painting was to be paid by the family, which explains uneven qualities in the portraits.
Click to view slideshow.
Oddly, a collection of stuffed animals keeps company with the pupils portraits, along with a collection of minerals.
Click to view slideshow.The place also used to host a weather station which collected some of the oldest known weather time series. Today, the rooftop observatory is one of the highlights of the place for visiting scholars with its breathtaking view over the city and the Alps in clear days.
Click to view slideshow.
Hi again!
In this post, I’ll explain the new smoother introduced in our paper Coupling of Particle Filters with Fredrik Lindsten and Thomas B. Schön from Uppsala University. Smoothing refers to the task of estimating a latent process of length , given noisy measurements of it, ; the smoothing distribution refers to . The setting is state-space models (what else?!), with a fixed parameter assumed to have been previously estimated.
Our smoother builds upon two recent innovations: the first is the conditional particle filter of Andrieu, Doucet and Holenstein (2010), and the second one is a debiasing technique of Glynn and Rhee (2014). The conditional particle filter (CPF) is a Markov kernel on the space of trajectories. The CPF kernel leaves the smoothing distribution invariant, so it can be iterated to obtain an MCMC sample approximating the smoothing distribution. On the other hand, the debiasing method takes a Markov kernel and a function and spits out an unbiased estimator of the integral of with respect to the invariant distribution of .
So here is the algorithm: we start with a pair of arbitrary trajectories, denoted by and . We first apply a step of CPF to , yielding (and we do nothing to ; bear with me!). Then we apply a “coupled CPF” kernel to the pair , yielding a new pair . What is a coupled CPF? It’s essentially a CPF kernel applied to both trajectories, using common random numbers and with a fancy resampling scheme as alluded to in the last post.
Then we iterate the coupled CPF kernel, yielding pairs , , etc. It’s just a Markov chain on the space of pairs of trajectories… BUT! at some step , the two trajectories become identical: . Tadaaaah! See the figure above, where the two trajectories meet in a few steps. And once the two trajectories meet, they stay together forever after (so cute). This is exciting because it means that (just adding infinitely many zeros). Now it is easy to see that this infinite sum has expectation given precisely by . Indeed, provided that we can swap limit and expectation, the expectation of the sum is . Since by the coupling construction, the sum is telescopic and we are left with , simply because the CPF chain itself is ergodic.
What’s the point? Instead of running a long MCMC chain, staring blankly at the chain to decide how many iterations are enough iterations, the above scheme can be run times completely in parallel. Each run takes a random but small number of steps, and produces an unbiased smoothing estimator. We can then average these estimators to get the final result, with accurate confidence intervals provided by our good friend the central limit theorem.
Thanks for reading, you deserve a Santana song on smoothing.
In this post, I’ll write about coupling particle filters, as proposed in our recent paper with Fredrik Lindsten and Thomas B. Schön from Uppsala University, available on arXiv; and also in this paper by colleagues at NUS. The paper is about a methodology with multiple direct consequences. In this first post, I’ll focus on correlated likelihood estimators; in a later post, I’ll describe a new smoothing algorithm. Both are described in detail in the article. We’ve been blessed to have been advertised by xi’an’s og, so glory is just around the corner.
Everybody is interested in estimating the likelihood , for some parameter , in general state space models (right?). These likelihoods are typically approximated by algorithms called particle filters. They take as input and spit out a likelihood estimator . I ran particle filters five times for many parameters , and plotted the resulting estimators in the following figure. The red curve indicates the exact log-likelihood.
These estimators are not so great! They all underestimate the log-likelihood by a large amount. Why? Here the model has a five-dimensional latent process over time steps, so that the likelihood is effectively defined by a -dimensional integral. Since I’ve used only particles in the filter, I obtain poor results. A first solution would be to use more particles, but the algorithmic complexity increases (linearly) with . If the maximum number of particles that I can use is , what can I do?
Suppose that in fact, we are interested in comparing two different likelihood values. This is a common task: think for instance of likelihood ratios in Metropolis-Hastings acceptance ratios. To make the estimator of a likelihood ratio more precise, we can introduce dependencies between the numerator and the denominator; an old variance reduction trick! More precisely, for two parameters and , we can consider correlated likelihood estimators and , such that, if over/under-estimates , then also over/under-estimates , so that overall the ratio can be more accurately estimated. This has been attempted many times in particle filtering, one of the first attempts being Mike Pitt’s. The basic idea is to use common random numbers for both particle filters, and then to fiddle with the resampling step, where the main difficulty lies. Intrinsically, the resampling step is a discrete operator that introduces discontinuities in the likelihood estimator, as a function of .
Following Mike Pitt and other works, we propose new resampling schemes related to optimal transport and maximal coupling ideas. The new schemes result in estimators of the likelihood as shown in the following figure.
Note that we are still performing poorly in absolute terms, but now the comparison between two likelihood estimators for nearby values of is more faithful to the true likelihood. This turns out to have pretty drastic effects in the estimation of the score function, or in Metropolis-Hastings schemes such as the correlated pseudo-marginal algorithm.
My last post dates back to May 2015… thanks to JB and Julyan for keeping the place busy! I’m not (quite) dead and intend to go back to posting stuff every now and then. And by the way, congrats to both for their new jobs!
Last July, I’ve also started a new job, as an assistant professor in the Department of Statistics at Harvard University, after having spent two years in Oxford. At some point, I might post something on the cultural difference between the European English and American communities of statisticians.
In the coming weeks, I’ll tell you all about a new paper entitled Coupling of Particle Filters, co-written with Fredrik Lindsten and Thomas B. Schön from Uppsala University in Sweden. We are excited about this coupling idea because it’s simple and yet brings massive gains in many important aspects of inference for state space models (including both parameter inference and smoothing). I’ll be talking about it at the World Congress in Probability and Statistics in Toronto next week and at JSM in Chicago, early in August.
I’ll also try to write about another exciting project, joint work with Christian Robert, Chris Holmes and Lawrence Murray, on modularization, cutting feedback, the infamous cut function of BUGS and all that funny stuff. I’ve talked about it in ISBA 2016, and intend to put the associated tech report on arXiv over the summer.
Stay tuned!