Bayesian models have been used extensively in various machine learning tasks, often resulting in improved prediction performance through the utilization of (layers of) latent variables when modeling the generative process of the observed data. Extending the parameter space from finite to infinite-dimensional, Bayesian nonparametric models can infer the model complexity directly from the data and thus also adapt with the amount of the observed data. This is especially appealing in the age of big data. However, such benefits come at a price: the parameter training and the prediction are notoriously difficult even for parametric models. Sampling and variational inference techniques are two standard methods for inference in Bayesian models, but for many problems, neither approach scales effectively to large-scale data. Currently, there is significant ongoing research trying to scale these methods using ideas from stochastic differential equations and stochastic optimization. A recent thread of research has considered small-variance asymptotics of latent-variable models as a way to capture the benefits of rich probabilistic models while also providing a framework for designing more scalable combinatorial optimization algorithms. Such models are often motivated by the well-known connection between mixtures of Gaussians and K-means: as the variances of the Gaussians tend to zero, the mixture of Gaussians model approaches K-means, both in terms of objectives and algorithms. In this dissertation, we will study small-variance asymptotics of Bayesian models, yielding new formulations and algorithms which may provide more efficient solutions to various unsupervised learning problems. Firstly, we consider clustering problems: exploring small-variance asymptotics for exponential family Dirichlet process (DP) and hierarchical Dirichlet process (HDP) mixture models. Utilizing connections between exponential family distributions and Bregman divergences, we derive novel clustering algorithms from the asymptotic limit of the DP and HDP mixtures that features the scalability of existing hard clustering methods as well as the flexibility of Bayesian nonparametric models. Secondly, we consider sequential models: exploring the small-variance asymptotic analysis of the infinite hidden Markov models, yielding a combinatorial objective function for discrete-data sequence observations with a non-fixed number of states. This involves a k-means-like term along with penalties based on state transitions and the number of states. We also present a simple, scalable, and flexible algorithm to optimize it. Lastly, we consider the topic modeling problems, which have emerged as fundamental tools in unsupervised machine learning. We approach it via combinatorial optimization, and take a small-variance limit of the latent Dirichlet allocation model to derive a new objective function. We minimize this objective by using ideas from combinatorial optimization, obtaining a new, fast, and high-quality topic modeling algorithm. In particular, we show that our results are not only significantly better than traditional small-variance asymptotic based algorithms, but also truly competitive with popular probabilistic approaches.

Abstract: Clustering of data has been a very well studied topic in the Machine Learning community, with various different methods trying to solve the same problem of grouping similar objects together. Traditional approaches have been algorithmically simpler and easier to implement with reasonable results. More recently, algorithms derived from asymptotics on Bayesian Non-parametric Infinite Mixture Models have appeared as an alternate. These algorithms in general have pointed at a very clear relation between probabilistic methods like Expectation Maximization, and hard assignment based algorithms like K-Means. They provide both the flexibility of a Bayesian Non-parametric model and scalability of hard clustering algorithms like K-Means. Aysmptotics on further complex mixture models have been used to derive algorithmsthat resemble hierarchical clustering and hard Topic Modeling.

This book gathers the outcomes of the thirteenth Workshop on the Algorithmic Foundations of Robotics (WAFR), the premier event for showcasing cutting-edge research on algorithmic robotics. The latest WAFR, held at Universidad Politécnica de Yucatán in Mérida, México on December 9–11, 2018, continued this tradition. This book contains fifty-four papers presented at WAFR, which highlight the latest research on fundamental algorithmic robotics (e.g., planning, learning, navigation, control, manipulation, optimality, completeness, and complexity) demonstrated through several applications involving multi-robot systems, perception, and contact manipulation. Addressing a diverse range of topics in papers prepared by expert contributors, the book reflects the state of the art and outlines future directions in the field of algorithmic robotics.

Now in its third edition, this classic book is widely considered the leading text on Bayesian methods, lauded for its accessible, practical approach to analyzing data and solving research problems. Bayesian Data Analysis, Third Edition continues to take an applied approach to analysis using up-to-date Bayesian methods. The authors—all leaders in the statistics community—introduce basic concepts from a data-analytic perspective before presenting advanced methods. Throughout the text, numerous worked examples drawn from real applications and research emphasize the use of Bayesian inference in practice. New to the Third Edition Four new chapters on nonparametric modeling Coverage of weakly informative priors and boundary-avoiding priors Updated discussion of cross-validation and predictive information criteria Improved convergence monitoring and effective sample size calculations for iterative simulation Presentations of Hamiltonian Monte Carlo, variational Bayes, and expectation propagation New and revised software code The book can be used in three different ways. For undergraduate students, it introduces Bayesian inference starting from first principles. For graduate students, the text presents effective current approaches to Bayesian modeling and computation in statistics and related fields. For researchers, it provides an assortment of Bayesian methods in applied statistics. Additional materials, including data sets used in the examples, solutions to selected exercises, and software instructions, are available on the book’s web page.

This book reviews nonparametric Bayesian methods and models that have proven useful in the context of data analysis. Rather than providing an encyclopedic review of probability models, the book’s structure follows a data analysis perspective. As such, the chapters are organized by traditional data analysis problems. In selecting specific nonparametric models, simpler and more traditional models are favored over specialized ones. The discussed methods are illustrated with a wealth of examples, including applications ranging from stylized examples to case studies from recent literature. The book also includes an extensive discussion of computational methods and details on their implementation. R code for many examples is included in online software pages.