Probabilistic graphical models such as Markov random fields provide a powerful framework and tools for machine learning, especially for structured output learning. Latent variables naturally exist in many applications of these models; they may arise from partially labeled data, or be introduced to enrich model flexibility. However, the presence of latent variables presents challenges for learning and inference.For example, the standard approach of using maximum a posteriori (MAP) prediction is complicated by the fact that, in latent variable models (LVMs), we typically want to first marginalize out the latent variables, leading to an inference task called marginal MAP. Unfortunately, marginal MAP prediction can be NP-hard even on relatively simple models such as trees, and few methods have been developed in the literature. This thesis presents a class of variational bounds for marginal MAP that generalizes the popular dual-decomposition method for MAP inference, and enables an efficient block coordinate descent algorithm to solve the corresponding optimization. Similarly, when learning LVMs for structured prediction, it is critically important to maintain the effect of uncertainty over latent variables by marginalization. We propose the marginal structured SVM, which uses marginal MAP inference to properly handle that uncertainty inside the framework of max-margin learning.We then turn our attention to an important subclass of latent variable models, restricted Boltzmann machines (RBMs). RBMs are two-layer latent variable models that are widely used to capture complex distributions of observed data, including as building block for deep probabilistic models. One practical problem in RBMs is model selection: we need to determine the hidden (latent) layer size before performing learning. We propose an infinite RBM model and apply the Frank-Wolfe algorithm to solve the resulting learning problem. The resulting algorithm can be interpreted as inserting a hidden variable into a RBM model at each iteration, to easily and efficiently perform model selection during learning. We also study the role of approximate inference in RBMs and conditional RBMs. In particular, there is a common assumption that belief propagation methods do not work well on RBM-based models, especially for learning. In contrast, we demonstrate that for conditional models and structured prediction, learning RBM-based models with belief propagation and its variants can provide much better results than the state-of-the-art contrastive divergence methods.

The current volume, Advances in Latent Variable Mixture Models, contains chapters by all of the speakers who participated in the 2006 CILVR conference, providing not just a snapshot of the event, but more importantly chronicling the state of the art in latent variable mixture model research. The volume starts with an overview chapter by the CILVR conference keynote speaker, Bengt Muthén, offering a “lay of the land” for latent variable mixture models before the volume moves to more specific constellations of topics. Part I, Multilevel and Longitudinal Systems, deals with mixtures for data that are hierarchical in nature either due to the data’s sampling structure or to the repetition of measures (of varied types) over time. Part II, Models for Assessment and Diagnosis, addresses scenarios for making judgments about individuals’ state of knowledge or development, and about the instruments used for making such judgments. Finally, Part III, Challenges in Model Evaluation, focuses on some of the methodological issues associated with the selection of models most accurately representing the processes and populations under investigation. It should be stated that this volume is not intended to be a first exposure to latent variable methods. Readers lacking such foundational knowledge are encouraged to consult primary and/or secondary didactic resources in order to get the most from the chapters in this volume. Once armed with the basic understanding of latent variable methods, we believe readers will find this volume incredibly exciting.

We present a family of probabilistic generative models that encompasses a variety of probability distributions (including Gaussian, Gamma, Beta, Poisson and many more) from the exponential family. In addition, we investigate a point-wise maximum function and introduce a novel non-linear superposition for coupling the latents and observables using two matrices (if the considered noise distribution has two parameters): One to model the component means and another for component variances. We further exploit the Expectation Maximization (EM) algorithm and show that the presented link function allows for the derivation of a very general and concise set of parameter update equations. Concretely, we derive a set of updates that have the same functional form for all regular distributions of the exponential family. Our results then provide directly applicable learning equations for commonly as well as for unusually distributed data. Finally, to assess the reliability of our theoretical findings, we consider different applications of the proposed generative models and investigate a variety of complex datasets including both synthetic and real data.

This volume gathers refereed papers presented at the 1994 UCLA conference on "La tent Variable Modeling and Application to Causality. " The meeting was organized by the UCLA Interdivisional Program in Statistics with the purpose of bringing together a group of people who have done recent advanced work in this field. The papers in this volume are representative of a wide variety of disciplines in which the use of latent variable models is rapidly growing. The volume is divided into two broad sections. The first section covers Path Models and Causal Reasoning and the papers are innovations from contributors in disciplines not traditionally associated with behavioural sciences, (e. g. computer science with Judea Pearl and public health with James Robins). Also in this section are contri butions by Rod McDonald and Michael Sobel who have a more traditional approach to causal inference, generating from problems in behavioural sciences. The second section encompasses new approaches to questions of model selection with emphasis on factor analysis and time varying systems. Amemiya uses nonlinear factor analysis which has a higher order of complexity associated with the identifiability condi tions. Muthen studies longitudinal hierarchichal models with latent variables and treats the time vector as a variable rather than a level of hierarchy. Deleeuw extends exploratory factor analysis models by including time as a variable and allowing for discrete and ordi nal latent variables. Arminger looks at autoregressive structures and Bock treats factor analysis models for categorical data.