One of the important problems in machine learning is determining the complexity of the model to learn. Too much complexity leads to overfitting, which finds structures that do not actually exist in the data, while too low complexity leads to underfitting, which means that the expressiveness of the model is insufficient to capture all the structures present in the data. For some probabilistic models, the complexity depends on the introduction of one or more latent variables whose role is to explain the generative process of the data. There are various approaches to identify the appropriate number of latent variables of a model. This thesis covers various Bayesian nonparametric methods capable of determining the number of latent variables to be used and their dimensionality. The popularization of Bayesian nonparametric statistics in the machine learning community is fairly recent. Their main attraction is the fact that they offer highly flexible models and their complexity scales appropriately with the amount of available data. In recent years, research on Bayesian nonparametric learning methods have focused on three main aspects: the construction of new models, the development of inference algorithms and new applications. This thesis presents our contributions to these three topics of research in the context of learning latent variables models. Firstly, we introduce the Pitman-Yor process mixture of Gaussians, a model for learning infinite mixtures of Gaussians. We also present an inference algorithm to discover the latent components of the model and we evaluate it on two practical robotics applications. Our results demonstrate that the proposed approach outperforms, both in performance and flexibility, the traditional learning approaches. Secondly, we propose the extended cascading Indian buffet process, a Bayesian nonparametric probability distribution on the space of directed acyclic graphs. In the context of Bayesian networks, this prior is used to identify the presence of latent variables and the network structure among them. A Markov Chain Monte Carlo inference algorithm is presented and evaluated on structure identification problems and as well as density estimation problems. Lastly, we propose the Indian chefs process, a model more general than the extended cascading Indian buffet process for learning graphs and orders. The advantage of the new model is that it accepts connections among observable variables and it takes into account the order of the variables. We also present a reversible jump Markov Chain Monte Carlo inference algorithm which jointly learns graphs and orders. Experiments are conducted on density estimation problems and testing independence hypotheses. This model is the first Bayesian nonparametric model capable of learning Bayesian learning networks with completely arbitrary graph structures.

Bayesian nonparametric methods have become increasingly popular in machine learning for their ability to allow the data to determine model complexity. In particular, Bayesian nonparametric versions of common latent variable models can learn as effective dimension of the latent space. Examples include mixture models, latent feature models and topic models, where the number of components, features, or topics need not be specified a priori. A drawback of many of these models is that they assume the observations are exchangeable, that is, any dependencies between observations are ignored. This thesis contributes general methods to incorporate covariates into Bayesian nonparametric models and inference algorithms to learn with these models. First, we will present a flexible class of dependent Bayesian nonparametric priors to induce covariate-dependence into a variety of latent variable models used in machine learning. The proposed framework has nice analytic properites and admits a simple inference algorithm. We show how the framework can be used to construct a covariate-dependent latent feature model and a time-varying topic model. Second, we describe the first general purpose inference algorithm for a large family of dependent mixture models. Using the idea of slice-sampling, the algorithm is truncation-free and fast, showing that inference can de done efficiently despite the added complexity that covariate-dependence entails. Last, we construct a Bayesian nonparametric framework to couple multiple latent variable models and apply the framework to learning from multiple views of data.

This book is the first systematic treatment of Bayesian nonparametric methods and the theory behind them. It will also appeal to statisticians in general. The book is primarily aimed at graduate students and can be used as the text for a graduate course in Bayesian non-parametrics.

Bayesian Nonparametrics via Neural Networks is the first book to focus on neural networks in the context of nonparametric regression and classification, working within the Bayesian paradigm. Its goal is to demystify neural networks, putting them firmly in a statistical context rather than treating them as a black box. This approach is in contrast to existing books, which tend to treat neural networks as a machine learning algorithm instead of a statistical model. Once this underlying statistical model is recognized, other standard statistical techniques can be applied to improve the model. The Bayesian approach allows better accounting for uncertainty. This book covers uncertainty in model choice and methods to deal with this issue, exploring a number of ideas from statistics and machine learning. A detailed discussion on the choice of prior and new noninformative priors is included, along with a substantial literature review. Written for statisticians using statistical terminology, Bayesian Nonparametrics via Neural Networks will lead statisticians to an increased understanding of the neural network model and its applicability to real-world problems.

Bayesian models have been used extensively in various machine learning tasks, often resulting in improved prediction performance through the utilization of (layers of) latent variables when modeling the generative process of the observed data. Extending the parameter space from finite to infinite-dimensional, Bayesian nonparametric models can infer the model complexity directly from the data and thus also adapt with the amount of the observed data. This is especially appealing in the age of big data. However, such benefits come at a price: the parameter training and the prediction are notoriously difficult even for parametric models. Sampling and variational inference techniques are two standard methods for inference in Bayesian models, but for many problems, neither approach scales effectively to large-scale data. Currently, there is significant ongoing research trying to scale these methods using ideas from stochastic differential equations and stochastic optimization. A recent thread of research has considered small-variance asymptotics of latent-variable models as a way to capture the benefits of rich probabilistic models while also providing a framework for designing more scalable combinatorial optimization algorithms. Such models are often motivated by the well-known connection between mixtures of Gaussians and K-means: as the variances of the Gaussians tend to zero, the mixture of Gaussians model approaches K-means, both in terms of objectives and algorithms. In this dissertation, we will study small-variance asymptotics of Bayesian models, yielding new formulations and algorithms which may provide more efficient solutions to various unsupervised learning problems. Firstly, we consider clustering problems: exploring small-variance asymptotics for exponential family Dirichlet process (DP) and hierarchical Dirichlet process (HDP) mixture models. Utilizing connections between exponential family distributions and Bregman divergences, we derive novel clustering algorithms from the asymptotic limit of the DP and HDP mixtures that features the scalability of existing hard clustering methods as well as the flexibility of Bayesian nonparametric models. Secondly, we consider sequential models: exploring the small-variance asymptotic analysis of the infinite hidden Markov models, yielding a combinatorial objective function for discrete-data sequence observations with a non-fixed number of states. This involves a k-means-like term along with penalties based on state transitions and the number of states. We also present a simple, scalable, and flexible algorithm to optimize it. Lastly, we consider the topic modeling problems, which have emerged as fundamental tools in unsupervised machine learning. We approach it via combinatorial optimization, and take a small-variance limit of the latent Dirichlet allocation model to derive a new objective function. We minimize this objective by using ideas from combinatorial optimization, obtaining a new, fast, and high-quality topic modeling algorithm. In particular, we show that our results are not only significantly better than traditional small-variance asymptotic based algorithms, but also truly competitive with popular probabilistic approaches.