Schedule for: 22w5092 - Advances in Stein’s method and its applications in Machine Learning and Optimization

A buffet dinner is served daily between 5:30pm and 7:30pm in Vistas Dining Room, top floor of the Sally Borden Building.

Breakfast is served daily between 7 and 9am in the Vistas Dining Room, the top floor of the Sally Borden Building.

A brief introduction to BIRS with important logistical information, technology instruction, and opportunity for participants to ask questions.

Sampling and optimization are fundamental tasks of machine learning. While the literature on optimization for machine learning has developed widely in the past decade, with fine convergence rates for some methods, the literature on sampling remained mainly asymptotic until very recently. Stein Variational Gradient Descent (SVGD) algorithm is a sampling algorithm that was derived in 2016 by Liu & Wang by taking advantage of a "kernelized" version of Stein's method. Besides, SVGD can be seen as an optimization algorithm over a space of probability measures to minimize the Kullback-Leibler divergence w.r.t. the target distribution. I will review these two points of view on SVGD, and show how they are equivalent through the prism of differential calculus over the Wasserstein space. In particular, this analogy allows to provide a quantitative convergence rate for SVGD in the so called population limit.

Introduced by I. Nourdin and G. Peccati in 2009, the "Malliavin-Stein method" is a collection of probabilistic techniques, allowing one to derive explicit analytic bounds on the normal and non-normal approximation of smooth functionals of Gaussian fields or point processes, by means of infinite-dimensional integration by parts formulae. Originally introduced in order to quantitatively study the fluctuations of Gaussian-subordinated sequences, the scope of applications of the Malliavin-Stein method has never ceased to grow - and now touches domains as diverse as computer sciences, stochastic geometry, mathematical statistics, mathematical physics, and compressed sensing. In this talk, I will introduce the main elements of the Malliavin-Stein method, and describe some distinguished applications - with special emphasis on functional estimates related to the classical Poincaré and log-Sobolev inequalities.

VC cheatsheet: https://docs.google.com/document/d/1vv9wFidXrbOuP2gdrOV_zujbBp7qjZTvz3YruMUtsio/edit VC link: https://link.virtualchair.net/birs/QXKvhx69xSvAA2Syg3qj/join

Lunch is served daily between 11:30am and 1:30pm in the Vistas Dining Room, the top floor of the Sally Borden Building.

As a statistical paradigm, Bayesian inference is conceptually simple and elegant. However, the computational challenge of sampling from the posterior distribution represents a major practical restriction on the class of models that can be analysed. Stein's method has recently emerged as a powerful tool in computational statistics, being used to construct novel sampling methods that have, in certain situations, out-performed the state-of-the-art. This tutorial will explain how Stein's method can be used to transform a sampling problem into an optimisation problem, before introducing several different optimisation algorithms that each give rise to a practical computational method for Bayesian inference.

Meet in foyer of TCPL to participate in the BIRS group photo. The photograph will be taken outdoors, so dress appropriately for the weather. Please don't be late, or you might not be in the official group photo!

Meet in the PDC front desk for a guided tour of The Banff Centre campus.

A buffet dinner is served daily between 5:30pm and 7:30pm in Vistas Dining Room, top floor of the Sally Borden Building.

Breakfast is served daily between 7 and 9am in the Vistas Dining Room, the top floor of the Sally Borden Building.

In this talk, I will explain how approximate optimizers in certain variational problems involving measures satisfy approximate integration-by-parts formulas. As a consequence, we can use Stein's method to show that approxmate minimizers are close to actual minimizers in some sense. This method will be illustrated by several examples of applications to sharp concentration inequalities and to Riemannian geometry.

In this talk, we will present a regularized formulation of the SVGD algorithm and present several results in the mean-filed limit setting. We start with a motivation for the proposed regularization and highlight how the regularized SVGD formulation relates to the Langevin diffusion. Next, we provide convergence results for the regularized SVGD in both the continuous-time and discrete-time setting (with the space being continuous in both cases). Finally, we provide existence and stability results for the non-linear PDE that arises as the mean-field limit of the regularized SVGD formulation.

Lunch is served daily between 11:30am and 1:30pm in the Vistas Dining Room, the top floor of the Sally Borden Building.

A buffet dinner is served daily between 5:30pm and 7:30pm in Vistas Dining Room, top floor of the Sally Borden Building.

Breakfast is served daily between 7 and 9am in the Vistas Dining Room, the top floor of the Sally Borden Building.

We establish multivariate second order Poincar\'e inequalities for Poisson functionals. The bounds are expressed as integrated moments of first and second order difference operators. These general results are shown to provide rates of multivariate normal convergence for a large class of vectors $(H_s^{(1)},\hdots,H_s^{(m)})$, $s \geq 1$, of statistics of marked Poisson processes on $\R^d$, $d \geq 2$, as the intensity parameter $s$ tends to infinity. The results are applicable whenever the functionals $H_s^{(i)}$, $i\in\{1,\hdots,m\}$, are expressible as sums of exponentially stabilizing score functions satisfying a moment condition. The rates are for the $d_2$-, $d_3$-, and $d_{convex}$-distances and are in general unimprovable.

Accurately approximating an unnormalized distribution with a discrete sample is a fundamental challenge in machine learning and statistical inference. Particle evolution methods like Stein variational gradient descent tackle this challenge by applying deterministic updates to particles to sequentially minimize Kullback-Leibler divergence. However, these methods break down for constrained targets and fails to exploit informative non-Euclidean geometry. In this talk, I will introduce a new family of particle evolution samplers suitable for constrained domains and non-Euclidean geometries. These samplers are derived from a new class of Stein operators and have deep connections with Riemannian Langevin diffusion, mirror descent, and natural gradient descent. We demonstrate that these new samplers yield accurate approximations to distributions on the simplex, deliver valid confidence intervals in post-selection inference, and converge more rapidly than prior methods in large-scale unconstrained posterior inference. Finally, we establish the convergence of our new procedures under verifiable conditions on the target distribution.

Lunch is served daily between 11:30am and 1:30pm in the Vistas Dining Room, the top floor of the Sally Borden Building.

A buffet dinner is served daily between 5:30pm and 7:30pm in Vistas Dining Room, top floor of the Sally Borden Building.

Breakfast is served daily between 7 and 9am in the Vistas Dining Room, the top floor of the Sally Borden Building.

The kernel Stein discrepancy (KSD) is a measure of fit between model and data, based on a Stein-modified kernel integral probability metric. The KSD is widely used in statistical tests of goodness-of-fit, since it does not require the normalising constant of the model to be known. I will describe a generalisation of the KSD, which incorporates models with latent variables, even when marginalisation over the latents is intractable. This generalised KSD is used in a relative goodness-of-fit test, where two alternative models are compared against the data: this reflects our understanding that "all models are wrong," but that some models are better than others. The relative test will be demonstrated in two settings: probabilistic PCA, and a Latent Dirichlet Allocation topic model. I will compare against the relative Maximum Mean Discrepancy test, which is based on samples from the models, and does not exploit the latent structure.

We study the first gradient step on the first-layer parameters $\boldsymbol{W}\in\R^{d \times N}$ of a width-N two-layer neural network, where the training objective is the empirical MSE loss: $\frac{1}{n}\sum_{i=1}^n (f(\boldsymbol{x}_i)-y_i)^2$. In the proportional asymptotic limit: $n,d,N\to\infty$ at the same rate, and an idealized student-teacher setting, we show that the first gradient update contains a rank-1 "spike", which results in the alignment between the first-layer weights and the linear component of the teacher model $f^*$. To characterize the impact of this alignment, we compute the prediction risk of ridge regression on the conjugate kernel after one gradient step on $\vW$ with learning rate $\eta$, when $f^*$ is a single-index model. We consider two scalings of the first step learning rate $\eta$. For small $\eta$, based on a recently established Stein-CLT in the feature space, we prove a Gaussian equivalence property for the trained feature map; this allows us to show that the learned kernel improves upon the initial random features model, but cannot defeat the best linear model on the input. Whereas for sufficiently large $\eta$, we prove that for certain $f^*$, the same ridge estimator on trained features can go beyond this "linear regime" and outperform a wide range of random features and rotationally invariant kernel models.

Lunch is served daily between 11:30am and 1:30pm in the Vistas Dining Room, the top floor of the Sally Borden Building.

A buffet dinner is served daily between 5:30pm and 7:30pm in Vistas Dining Room, top floor of the Sally Borden Building.

Breakfast is served daily between 7 and 9am in the Vistas Dining Room, the top floor of the Sally Borden Building.

5-day workshop participants are welcome to use BIRS facilities (TCPL ) until 3 pm on Friday, although participants are still required to checkout of the guest rooms by 11AM.