In this dissertation, we explore a class of unifying and scalable algorithms for a number of submodular optimization problems, and connect them to several machine learning applications. These optimization problems include, 1. Constrained and Unconstrained Submodular Minimization, 2. Constrained and Unconstrained Submodular Maximization, 3. Difference of Submodular Optimization 4. Submodular Optimization subject to Submodular Constraints The main focus of this thesis, is to study these problems theoretically, in the light of the machine learning problems where they naturally occur. We provide scalable, practical and unifying algorithms for all the above optimization problems, which retain good theoretical guarantees, and investigate the underlying hardness of these optimization problems. We also study natural subclasses of submodular functions, along with theoretical constructs, which help connect theory to practice by providing tighter worst case guarantees. While the focus of this thesis is mainly theoretical, we also empirically demonstrate the applicability of these techniques on several synthetic and real world problems.

Submodular functions are relevant to machine learning for at least two reasons: (1) some problems may be expressed directly as the optimization of submodular functions and (2) the Lovász extension of submodular functions provides a useful set of regularization functions for supervised and unsupervised learning. In this monograph, we present the theory of submodular functions from a convex analysis perspective, presenting tight links between certain polyhedra, combinatorial optimization and convex optimization problems. In particular, we show how submodular function minimization is equivalent to solving a wide variety of convex optimization problems. This allows the derivation of new efficient algorithms for approximate and exact submodular function minimization with theoretical guarantees and good practical performance. By listing many examples of submodular functions, we review various applications to machine learning, such as clustering, experimental design, sensor placement, graphical model structure learning or subset selection, as well as a family of structured sparsity-inducing norms that can be derived and used from submodular functions.

Learning with Submodular Functions presents the theory of submodular functions in a self-contained way from a convex analysis perspective, presenting tight links between certain polyhedra, combinatorial optimization and convex optimization problems.

This book presents a framework for the control of networked systems utilizing submodular optimization techniques. The main focus is on selecting input nodes for the control of networked systems, an inherently discrete optimization problem with applications in power system stability, social influence dynamics, and the control of vehicle formations. The first part of the book is devoted to background information on submodular functions, matroids, and submodular optimization, and presents algorithms for distributed submodular optimization that are scalable to large networked systems. In turn, the second part develops a unifying submodular optimization approach to controlling networked systems based on multiple performance and controllability criteria. Techniques are introduced for selecting input nodes to ensure smooth convergence, synchronization, and robustness to environmental and adversarial noise. Submodular optimization is the first unifying approach towards guaranteeing both performance and controllability with provable optimality bounds in static as well as time-varying networks. Throughout the text, the submodular framework is illustrated with the help of numerical examples and application-based case studies in biological, energy and vehicular systems. The book effectively combines two areas of growing interest, and will be especially useful for researchers in control theory, applied mathematics, networking or machine learning with experience in submodular optimization but who are less familiar with the problems and tools available for networked systems (or vice versa). It will also benefit graduate students, offering consistent terminology and notation that greatly reduces the initial effort associated with beginning a course of study in a new area.

It has widely been recognized that submodular functions play essential roles in efficiently solvable combinatorial optimization problems. Since the publication of the 1st edition of this book fifteen years ago, submodular functions have been showing further increasing importance in optimization, combinatorics, discrete mathematics, algorithmic computer science, and algorithmic economics, and there have been made remarkable developments of theory and algorithms in submodular functions. The 2nd edition of the book supplements the 1st edition with a lot of remarks and with new two chapters: "Submodular Function Minimization" and "Discrete Convex Analysis." The present 2nd edition is still a unique book on submodular functions, which is essential to students and researchers interested in combinatorial optimization, discrete mathematics, and discrete algorithms in the fields of mathematics, operations research, computer science, and economics. - Self-contained exposition of the theory of submodular functions - Selected up-to-date materials substantial to future developments - Polyhedral description of Discrete Convex Analysis - Full description of submodular function minimization algorithms - Effective insertion of figures - Useful in applied mathematics, operations research, computer science, and economics

The revolution in information technology and the explosion in the use of computing devices in people's everyday activities has forever changed the perspective of the data mining and machine learning fields. The enormous amounts of easily accessible, information rich data is pushing the data analysis community in general towards a shift of paradigm. In the new paradigm, data comes in the form a stream of billions of records received everyday. The dynamic nature of the data and its sheer size makes it impossible to use the traditional notion of offline learning where the whole data is accessible at any time point. Moreover, no amount of human resources is enough to get expert feedback on the data. In this work we have developed a unified optimization based learning framework that approaches many of the challenges mentioned earlier. Specifically, we developed a Never-Ending Learning framework which combines incremental/online, semi-supervised, and active learning under a unified optimization framework. The established framework is based on the class of submodular optimization methods. At the core of this work we provide a novel formulation of the Semi-Supervised Support Vector Machines (S3VM) in terms of submodular set functions. The new formulation overcomes the non-convexity issues of the S3VM and provides a state of the art solution that is orders of magnitude faster than the cutting edge algorithms in the literature. Next, we provide a stream summarization technique via exemplar selection. This technique makes it possible to keep a fixed size exemplar representation of a data stream that can be used by any label propagation based semi-supervised learning technique. The compact data steam representation allows a wide range of algorithms to be extended to incremental/online learning scenario. Under the same optimization framework, we provide an active learning algorithm that constitute the feedback between the learning machine and an oracle. Finally, the developed Never-Ending Learning framework is essentially transductive in nature. Therefore, our last contribution is an inductive incremental learning technique for incremental training of SVM using the properties of local kernels. We demonstrated through this work the importance and wide applicability of the proposed methodologies.

Active learning is a machine learning setting where the learning algorithm decides what data is labeled. Submodular functions are a class of set functions for which many optimization problems have efficient exact or approximate algorithms. We examine their connections. 1. We propose a new class of interactive submodular optimization problems which connect and generalize submodular optimization and active learning over a finite query set. We derive greedy algorithms with approximately optimal worst-case cost. These analyses apply to exact learning, approximate learning, learning in the presence of adversarial noise, and applications that mix learning and covering. 2. We consider active learning in a batch, transductive setting where the learning algorithm selects a set of examples to be labeled at once. In this setting we derive new error bounds which use symmetric submodular functions for regularization, and we give algorithms which approximately minimize these bounds. 3. We consider a repeated active learning setting where the learning algorithm solves a sequence of related learning problems. We propose an approach to this problem based on a new online prediction version of submodular set cover. A common theme in these results is the use of tools from submodular optimization to extend the breadth and depth of learning theory with an emphasis on non-stochastic settings.