This book is an introduction to optimal stochastic control for continuous time Markov processes and the theory of viscosity solutions. It covers dynamic programming for deterministic optimal control problems, as well as to the corresponding theory of viscosity solutions. New chapters in this second edition introduce the role of stochastic optimal control in portfolio optimization and in pricing derivatives in incomplete markets and two-controller, zero-sum differential games.

This book is concerned with a class of discrete-time stochastic control processes known as controlled Markov processes (CMP's), also known as Markov decision processes or Markov dynamic programs. Starting in the mid-1950swith Richard Bellman, many contributions to CMP's have been made, and applications to engineering, statistics and operations research, among other areas, have also been developed. The purpose of this book is to present some recent developments on the theory of adaptive CMP's, i. e. , CMP's that depend on unknown parameters. Thus at each decision time, the controller or decision-maker must estimate the true parameter values, and then adapt the control actions to the estimated values. We do not intend to describe all aspects of stochastic adaptive control; rather, the selection of material reflects our own research interests. The prerequisite for this book is a knowledgeof real analysis and prob ability theory at the level of, say, Ash (1972) or Royden (1968), but no previous knowledge of control or decision processes is required. The pre sentation, on the other hand, is meant to beself-contained,in the sensethat whenever a result from analysisor probability is used, it is usually stated in full and references are supplied for further discussion, if necessary. Several appendices are provided for this purpose. The material is divided into six chapters. Chapter 1 contains the basic definitions about the stochastic control problems we are interested in; a brief description of some applications is also provided.

This book is devoted to the systematic exposition of the contemporary theory of controlled Markov processes with discrete time parameter or in another termi nology multistage Markovian decision processes. We discuss the applications of this theory to various concrete problems. Particular attention is paid to mathe matical models of economic planning, taking account of stochastic factors. The authors strove to construct the exposition in such a way that a reader interested in the applications can get through the book with a minimal mathe matical apparatus. On the other hand, a mathematician will find, in the appropriate chapters, a rigorous theory of general control models, based on advanced measure theory, analytic set theory, measurable selection theorems, and so forth. We have abstained from the manner of presentation of many mathematical monographs, in which one presents immediately the most general situation and only then discusses simpler special cases and examples. Wishing to separate out difficulties, we introduce new concepts and ideas in the simplest setting, where they already begin to work. Thus, before considering control problems on an infinite time interval, we investigate in detail the case of the finite interval. Here we first study in detail models with finite state and action spaces-a case not requiring a departure from the realm of elementary mathematics, and at the same time illustrating the most important principles of the theory.

Great advances have been made in recent years in the field of computational probability. In particular, the state of the art - as it relates to queuing systems, stochastic Petri-nets and systems dealing with reliability - has benefited significantly from these advances. The objective of this book is to make these topics accessible to researchers, graduate students, and practitioners. Great care was taken to make the exposition as clear as possible. Every line in the book has been evaluated, and changes have been made whenever it was felt that the initial exposition was not clear enough for the intended readership. The work of major research scholars in this field comprises the individual chapters of Computational Probability. The first chapter describes, in nonmathematical terms, the challenges in computational probability. Chapter 2 describes the methodologies available for obtaining the transition matrices for Markov chains, with particular emphasis on stochastic Petri-nets. Chapter 3 discusses how to find transient probabilities and transient rewards for these Markov chains. The next two chapters indicate how to find steady-state probabilities for Markov chains with a finite number of states. Both direct and iterative methods are described in Chapter 4. Details of these methods are given in Chapter 5. Chapters 6 and 7 deal with infinite-state Markov chains, which occur frequently in queueing, because there are times one does not want to set a bound for all queues. Chapter 8 deals with transforms, in particular Laplace transforms. The work of Ward Whitt and his collaborators, who have recently developed a number of numerical methods for Laplace transform inversions, is emphasized in this chapter. Finally, if one wants to optimize a system, one way to do the optimization is through Markov decision making, described in Chapter 9. Markov modeling has found applications in many areas, three of which are described in detail: Chapter 10 analyzes discrete-time queues, Chapter 11 describes networks of queues, and Chapter 12 deals with reliability theory.

New up-to-date edition of this influential classic on Markov chains in general state spaces. Proofs are rigorous and concise, the range of applications is broad and knowledgeable, and key ideas are accessible to practitioners with limited mathematical background. New commentary by Sean Meyn, including updated references, reflects developments since 1996.

Aims to give to the reader the tools necessary to apply semi-Markov processes in real-life problems. The book is self-contained and, starting from a low level of probability concepts, gradually brings the reader to a deep knowledge of semi-Markov processes. Presents homogeneous and non-homogeneous semi-Markov processes, as well as Markov and semi-Markov rewards processes. The concepts are fundamental for many applications, but they are not as thoroughly presented in other books on the subject as they are here.

"This well-written book provides a clear and accessible treatment of the theory of discrete and continuous-time Markov chains, with an emphasis towards applications. The mathematical treatment is precise and rigorous without superfluous details, and the results are immediately illustrated in illuminating examples. This book will be extremely useful to anybody teaching a course on Markov processes." Jean-François Le Gall, Professor at Université de Paris-Orsay, France. Markov processes is the class of stochastic processes whose past and future are conditionally independent, given their present state. They constitute important models in many applied fields. After an introduction to the Monte Carlo method, this book describes discrete time Markov chains, the Poisson process and continuous time Markov chains. It also presents numerous applications including Markov Chain Monte Carlo, Simulated Annealing, Hidden Markov Models, Annotation and Alignment of Genomic sequences, Control and Filtering, Phylogenetic tree reconstruction and Queuing networks. The last chapter is an introduction to stochastic calculus and mathematical finance. Features include: The Monte Carlo method, discrete time Markov chains, the Poisson process and continuous time jump Markov processes. An introduction to diffusion processes, mathematical finance and stochastic calculus. Applications of Markov processes to various fields, ranging from mathematical biology, to financial engineering and computer science. Numerous exercises and problems with solutions to most of them