This thesis tackles three different problems in high-dimensional statistics. The first two parts of the thesis focus on estimation of sparse high-dimensional undirected graphical models under non-standard conditions, specifically, non-Gaussianity and missingness, when observations are continuous. To address estimation under non-Gaussianity, we propose a general framework involving augmenting the score matching losses introduced in Hyva ̈rinen [2005, 2007] with an l1-regularizing penalty. This method, which we refer to as regularized score matching, allows for computationally efficient treatment of Gaussian and non-Gaussian continuous exponential family models because the considered loss becomes a penalized quadratic and thus yields piecewise linear solution paths. Under suitable irrepresentability conditions and distributional assumptions, we show that regularized score matching generates consistent graph estimates in sparse high-dimensional settings. Through numerical experiments and an application to RNAseq data, we confirm that regularized score matching achieves state-of- the-art performance in the Gaussian case and provides a valuable tool for computationally efficient estimation in non-Gaussian graphical models. To address estimation of sparse high-dimensional undirected graphical models with missing observations, we propose adapting the regularized score matching framework by substituting in surrogates of relevant statistics to accommodate these circumstances, as in Loh and Wainwright [2012] and Kolar and Xing [2012]. For Gaussian and non-Gaussian continuous exponential family models, the use of these surrogates may result in a loss of semi-definiteness, and thus nonconvexity, in the objective. Nevertheless, under suitable distributional assumptions, the global optimum is close to the truth in matrix l1 norm with high probability in sparse high-dimensional settings. Furthermore, under the same set of assumptions, we show that the composite gradient descent algorithm we propose for minimizing the modified objective converges at a geometric rate to a solution close to the global optimum with high probability. The last part of the thesis moves away from undirected graphical models, and is instead concerned with inference in high-dimensional regression models. Specifically, we investigate how to construct asymptotically valid confidence intervals and p-values for the fixed effects in a high-dimensional linear mixed effect model. The framework we propose, largely founded on a recent work [Bu ̈hlmann, 2013], entails de-biasing a ‘naive’ ridge estimator. We show via numerical experiments that the method controls for Type I error in hypothesis testing and generates confidence intervals that achieve target coverage, outperforming competitors that assume observations are homogeneous when observations are, in fact, correlated within group.

A wide variety of problems that are encountered in different fields can be formulated as an inference problem. Common examples of such inference problems include estimating parameters of a model from some observations, inverse problems where an unobserved signal is to be estimated based on a given model and some measurements, or a combination of the two where hidden signals along with some parameters of the model are to be estimated jointly. For example, various tasks in machine learning such as image inpainting and super-resolution can be cast as an inverse problem over deep neural networks. Similarly, in computational neuroscience, a common task is to estimate the parameters of a nonlinear dynamical system from neuronal activities. Despite wide application of different models and algorithms to solve these problems, our theoretical understanding of how these algorithms work is often incomplete. In this work, we try to bridge the gap between theory and practice by providing theoretical analysis of three different estimation problems. First, we consider the problem of estimating the input and hidden layer signals in a given multi-layer stochastic neural network with all the signals being matrix valued. Various problems such as multitask regression and classification, and inverse problems that use deep generative priors can be modeled as inference problem over multi-layer neural networks. We consider different types of estimators for such problems and exactly analyze the performance of these estimators in a certain high-dimensional regime known as the large system limit. Our analysis allows us to obtain the estimation error of all the hidden signals in the deep neural network as expectations over low-dimensional random variables that are characterized via a set of equations called the state evolution. Next, we analyze the problem of estimating a signal from convolutional observations via ridge estimation. Such convolutional inverse problems arise naturally in several fields such as imaging and seismology. The shared weights of the convolution operator introduces dependencies in the observations that makes analysis of such estimators difficult. By looking at the problem in the Fourier domain and using results about Fourier transform of a class of random processes, we show that this problem can be reduced to analysis of multiple ordinary ridge estimators, one for each frequency. This allows us to write the estimation error of the ridge estimator as an integral that depends on the spectrum of the underlying random process that generates the input features. Finally, we conclude this work by considering the problem of estimating the parameters of a multi-dimensional autoregressive generalized linear model with discrete values. Such processes take a linear combination of the past outputs of the process as the mean parameter of a generalized linear model that generates the future values. The coefficients of the linear combination are the parameters of the model and we seek to estimate these parameters under the assumption that they are sparse. This model can be used for example to model the spiking activity of neurons. In this problem, we obtain a high-probability upper bound for the estimation error of the parameters. Our experiments further support these theoretical results.

Economic modeling in a data-rich environment is often challenging. To allow for enough flexibility and to model heterogeneity, models might have parameters with dimensionality growing with (or even much larger than) the sample size of the data. Learning these high-dimensional parameters requires new methodologies and theories. We consider three important high-dimensional models and propose novel methods for estimation and inference. Empirical applications in economics and finance are also studied. In Chapter 1, we consider high-dimensional panel data models (large cross sections and long time horizons) with interactive fixed effects and allow the covariate/slope coefficients to vary over time without any restrictions. The parameter of interest is the vector that contains all the covariate effects across time. This vector has dimensionality tending to infinity, potentially much faster than the cross-sectional sample size. We develop methods for the estimation and inference of this high-dimensional vector, i.e., the entire trajectory of time variation in covariate effects. We show that both the consistency of our estimator and the asymptotic accuracy of the proposed inference procedure hold uniformly in time. Our methodology can be applied to several important issues in econometrics, such as constructing confidence bands for the entire path of covariate coefficients across time, testing the time-invariance of slope coefficients and estimation and inference of patterns of time variations, including structural breaks and regime switching. An important feature of our method is that it provides inference procedures for the time variation in pre-specified components of slope coefficients while allowing for arbitrary time variation in other components. Computationally, our procedures do not require any numerical optimization and are very simple to implement. Monte Carlo simulations demonstrate favorable properties of our methods in finite samples. We illustrate our methods through empirical applications in finance and economics. In Chapter 2, we consider large factor models with unobserved factors. We formalize the notion of common factors between different groups of variables and propose to use it as a general approach to study the structure of factors, i.e., which factors drive which variables. The spanning hypothesis, which states that factors driving one group are spanned by those driving another group, can be studied as a special case under our framework. We develop a statistical procedure for testing the number of common factors. Our inference procedure is built upon recent results on high-dimensional bootstrap and is shown to be valid under the asymptotic framework of large $n$ and large $T$. In Monte Carlo simulations, our procedure performs well in finite samples. As an empirical application, we construct confidence sets for the number of common factors between the macroeconomy and the financial markets. Chapter 3 is joint work with Jelena Bradic. We propose a methodology for testing linear hypothesis in high-dimensional linear models. The proposed test does not impose any restriction on the size of the model, i.e. model sparsity or the loading vector representing the hypothesis. Providing asymptotically valid methods for testing general linear functions of the regression parameters in high-dimensions is extremely challenging--especially without making restrictive or unverifiable assumptions on the number of non-zero elements. We propose to test the moment conditions related to the newly designed restructured regression, where the inputs are transformed and augmented features. These new features incorporate the structure of the null hypothesis directly. The test statistics are constructed in such a way that lack of sparsity in the original model parameter does not present a problem for the theoretical justification of our procedures. We establish asymptotically exact control on Type I error without imposing any sparsity assumptions on model parameter or the vector representing the linear hypothesis. Our method is also shown to achieve certain optimality in detecting deviations from the null hypothesis. We demonstrate the favorable finite-sample performance of the proposed methods, via a number of numerical and a real data example.

In the last ten years, there has been increasing interest and activity in the general area of partially linear regression smoothing in statistics. Many methods and techniques have been proposed and studied. This monograph hopes to bring an up-to-date presentation of the state of the art of partially linear regression techniques. The emphasis is on methodologies rather than on the theory, with a particular focus on applications of partially linear regression techniques to various statistical problems. These problems include least squares regression, asymptotically efficient estimation, bootstrap resampling, censored data analysis, linear measurement error models, nonlinear measurement models, nonlinear and nonparametric time series models.

This article is about estimation and inference methods for high dimensional sparse (HDS) regression models in econometrics. High dimensional sparse models arise in situations where many regressors (or series terms) are available and the regression function is well-approximated by a parsimonious, yet unknown set of regressors. The latter condition makes it possible to estimate the entire regression function effectively by searching for approximately the right set of regressors. We discuss methods for identifying this set of regressors and estimating their coefficients based on l1 -penalization and describe key theoretical results. In order to capture realistic practical situations, we expressly allow for imperfect selection of regressors and study the impact of this imperfect selection on estimation and inference results. We focus the main part of the article on the use of HDS models and methods in the instrumental variables model and the partially linear model. We present a set of novel inference results for these models and illustrate their use with applications to returns to schooling and growth regression. -- inference under imperfect model selection ; structural effects ; high-dimensional econometrics ; instrumental regression ; partially linear regression ; returns-to-schooling ; growth regression

A graphical model is a statistical model that is represented by a graph. The factorization properties underlying graphical models facilitate tractable computation with multivariate distributions, making the models a valuable tool with a plethora of applications. Furthermore, directed graphical models allow intuitive causal interpretations and have become a cornerstone for causal inference. While there exist a number of excellent books on graphical models, the field has grown so much that individual authors can hardly cover its entire scope. Moreover, the field is interdisciplinary by nature. Through chapters by leading researchers from different areas, this handbook provides a broad and accessible overview of the state of the art. Key features: * Contributions by leading researchers from a range of disciplines * Structured in five parts, covering foundations, computational aspects, statistical inference, causal inference, and applications * Balanced coverage of concepts, theory, methods, examples, and applications * Chapters can be read mostly independently, while cross-references highlight connections The handbook is targeted at a wide audience, including graduate students, applied researchers, and experts in graphical models.

Statistical inference under high dimensional modelings has attracted much attention due to its wide applications in many fields. In this dissertation, I propose new methods for statistical inference in high dimensional models from three aspects: inference in high dimensional semiparametric models, inference in high dimensional matrix-valued data, and inference in high dimensional measurement error misspecified models. The first project studied statistical inference in high dimensional partially linear single index models. Firstly a profile partial penalized least squares estimator for parameter estimates for the model is proposed, and its asymptotic properties are given. Then an F-type test statistic for testing the parametric components is proposed, and its theoretical properties are established. I then propose a new test for the specification testing problem of the nonparametric components. Finally, simulation studies and empirical analysis of a real-world data set are conducted to illustrate the performance of the proposed testing procedure. The second project proposes new testing procedures in high dimensional matrix-valued data. Rank is an essential attribute for a matrix. A new type of statistic is proposed, which can make inferences on the rank of the matrix-valued data. I firstly give the theoretical property of its oracle version. To overcome the problem of empirical error accumulation, a new type of sparse SVD method is proposed, and its theoretical properties are given. Based on the newly proposed sparse SVD method, I provide a sample version statistic. Theoretical properties of this sample version statistic are given. Simulation studies and two applications to surveillance video data are provided to illustrate the performance of our newly proposed method. The third project proposes a new testing method in misspecified measurement error models. The testing method can work when there is potential model misspecification and measurement error in the model. Firstly its property is studied under the low dimensional setting. Then I develop it to the high dimensional setting. Further, I propose a method that can be adaptive to the sparsity level of the true parameters under the high dimensional setting. Simulation studies and one application to a clinical trial data set are given.