The work presents a modern, unified view on decision support and planning by considering its basics like preferences, belief, possibility and probability as well as utilities. These features together are immanent for software agents to believe the user that the agents are "intelligent".

Many real-world decision problems have multiple objectives. For example, when choosing a medical treatment plan, we want to maximize the efficacy of the treatment, but also minimize the side effects. These objectives typically conflict, e.g., we can often increase the efficacy of the treatment, but at the cost of more severe side effects. In this book, we outline how to deal with multiple objectives in decision-theoretic planning and reinforcement learning algorithms. To illustrate this, we employ the popular problem classes of multi-objective Markov decision processes (MOMDPs) and multi-objective coordination graphs (MO-CoGs). First, we discuss different use cases for multi-objective decision making, and why they often necessitate explicitly multi-objective algorithms. We advocate a utility-based approach to multi-objective decision making, i.e., that what constitutes an optimal solution to a multi-objective decision problem should be derived from the available information about user utility. We show how different assumptions about user utility and what types of policies are allowed lead to different solution concepts, which we outline in a taxonomy of multi-objective decision problems. Second, we show how to create new methods for multi-objective decision making using existing single-objective methods as a basis. Focusing on planning, we describe two ways to creating multi-objective algorithms: in the inner loop approach, the inner workings of a single-objective method are adapted to work with multi-objective solution concepts; in the outer loop approach, a wrapper is created around a single-objective method that solves the multi-objective problem as a series of single-objective problems. After discussing the creation of such methods for the planning setting, we discuss how these approaches apply to the learning setting. Next, we discuss three promising application domains for multi-objective decision making algorithms: energy, health, and infrastructure and transportation. Finally, we conclude by outlining important open problems and promising future directions.

Prescriptive Bayesian decision making has reached a high level of maturity and is well-supported algorithmically. However, experimental data shows that real decision makers choose such Bayes-optimal decisions surprisingly infrequently, often making decisions that are badly sub-optimal. So prevalent is such imperfect decision-making that it should be accepted as an inherent feature of real decision makers living within interacting societies. To date such societies have been investigated from an economic and gametheoretic perspective, and even to a degree from a physics perspective. However, little research has been done from the perspective of computer science and associated disciplines like machine learning, information theory and neuroscience. This book is a major contribution to such research. Some of the particular topics addressed include: How should we formalise rational decision making of a single imperfect decision maker? Does the answer change for a system of imperfect decision makers? Can we extend existing prescriptive theories for perfect decision makers to make them useful for imperfect ones? How can we exploit the relation of these problems to the control under varying and uncertain resources constraints as well as to the problem of the computational decision making? What can we learn from natural, engineered, and social systems to help us address these issues?

The purpose of this book is to develop in greater depth some of the methods from the author's Reinforcement Learning and Optimal Control recently published textbook (Athena Scientific, 2019). In particular, we present new research, relating to systems involving multiple agents, partitioned architectures, and distributed asynchronous computation. We pay special attention to the contexts of dynamic programming/policy iteration and control theory/model predictive control. We also discuss in some detail the application of the methodology to challenging discrete/combinatorial optimization problems, such as routing, scheduling, assignment, and mixed integer programming, including the use of neural network approximations within these contexts. The book focuses on the fundamental idea of policy iteration, i.e., start from some policy, and successively generate one or more improved policies. If just one improved policy is generated, this is called rollout, which, based on broad and consistent computational experience, appears to be one of the most versatile and reliable of all reinforcement learning methods. In this book, rollout algorithms are developed for both discrete deterministic and stochastic DP problems, and the development of distributed implementations in both multiagent and multiprocessor settings, aiming to take advantage of parallelism. Approximate policy iteration is more ambitious than rollout, but it is a strictly off-line method, and it is generally far more computationally intensive. This motivates the use of parallel and distributed computation. One of the purposes of the monograph is to discuss distributed (possibly asynchronous) methods that relate to rollout and policy iteration, both in the context of an exact and an approximate implementation involving neural networks or other approximation architectures. Much of the new research is inspired by the remarkable AlphaZero chess program, where policy iteration, value and policy networks, approximate lookahead minimization, and parallel computation all play an important role.

Multiagent systems is an expanding field that blends classical fields like game theory and decentralized control with modern fields like computer science and machine learning. This monograph provides a concise introduction to the subject, covering the theoretical foundations as well as more recent developments in a coherent and readable manner. The text is centered on the concept of an agent as decision maker. Chapter 1 is a short introduction to the field of multiagent systems. Chapter 2 covers the basic theory of singleagent decision making under uncertainty. Chapter 3 is a brief introduction to game theory, explaining classical concepts like Nash equilibrium. Chapter 4 deals with the fundamental problem of coordinating a team of collaborative agents. Chapter 5 studies the problem of multiagent reasoning and decision making under partial observability. Chapter 6 focuses on the design of protocols that are stable against manipulations by self-interested agents. Chapter 7 provides a short introduction to the rapidly expanding field of multiagent reinforcement learning. The material can be used for teaching a half-semester course on multiagent systems covering, roughly, one chapter per lecture.

Data-driven decision making, lying at the intersection between learning and decision making, has emerged as an important paradigm for engineering, data science and operations research at large. This thesis considers one facet of data-driven decision making, where multiple agents engage in an online learning process and make sequential decisions using data that become available over time. More specifically, we consider a model of multi-agent online strategic decision making, in which the reward structures of agents are given by a general continuous game and the feedback information to each agent is imperfect in one or more ways: each agent's feedback may suffer from some combination of noise corruption, delays and loss. The thesis presents an in-depth inquiry into the last-iterate convergence to Nash equilibria in the presence of such imperfect feedback, when each agent utilizes a no-regret online learning algorithm to maximize its cumulative performance. Last-iterate convergence (i.e. convergence of the actual joint action of all agents) stands in contrast with the more traditionally-studied time-average convergence in the existing literature (i.e. convergence of the time-average of the historical joint actions) and provides a more relevant (albeit more challenging) metric for online decision making problems. Unfortunately, last-iterate convergence (particularly when imperfect feedback is present) is under-explored in the existing work in multi-agent online learning. Rising to this challenge, this thesis aims to bridge the existing gap by answering some of the open questions in this field. In particular, a key high-level insight this thesis aims to articulate is that a broad family of no-regret learning algorithms, known as online mirror descent, can be adapted in multi-agent learning to guarantee last-iterate convergence to Nash equilibria in a general class of games under severely imperfect feedback information. Further, using power control in wireless networks as a motivating application domain, this thesis then harnesses these adapted online learning algorithms and theoretical convergence results to design robust and low-overhead distributed algorithms that operate in realistic environments and that come with strong performance guarantees. In sum, this thesis contributes to the broad landscape of multi-agent online learning by, among other things, making clear that the ambitious agenda of last-iterate convergence is not out of reach and should be the new norm (rather than the exception) for judging an algorithm's performance. A second (at least equally important) perspective that the thesis contributes pertains to distributed algorithm design in control and/or optimization: the often more complex process of developing robust distributed algorithms to achieve a desired/optimal system state can be transformed into the static and often simpler process of designing a game. With this transformation, all the algorithms and convergence results in multi-agent online learning can be immediately harnessed to yield practical algorithms for the problem at hand. This viewpoint has the potential to simplify the algorithm designer's task, whatever domain it may be.