Markov Chains are widely used as stochastic models to study a broad spectrum of system performance and dependability characteristics. This monograph is devoted to compositional specification and analysis of Markov chains. Based on principles known from process algebra, the author systematically develops an algebra of interactive Markov chains. By presenting a number of distinguishing results, of both theoretical and practical nature, the author substantiates the claim that interactive Markov chains are more than just another formalism: Among other, an algebraic theory of interactive Markov chains is developed, devise algorithms to mechanize compositional aggregation are presented, and state spaces of several million states resulting from the study of an ordinary telefone system are analyzed.

A fascinating and instructive guide to Markov chains for experienced users and newcomers alike This unique guide to Markov chains approaches the subject along the four convergent lines of mathematics, implementation, simulation, and experimentation. It introduces readers to the art of stochastic modeling, shows how to design computer implementations, and provides extensive worked examples with case studies. Markov Chains: From Theory to Implementation and Experimentation begins with a general introduction to the history of probability theory in which the author uses quantifiable examples to illustrate how probability theory arrived at the concept of discrete-time and the Markov model from experiments involving independent variables. An introduction to simple stochastic matrices and transition probabilities is followed by a simulation of a two-state Markov chain. The notion of steady state is explored in connection with the long-run distribution behavior of the Markov chain. Predictions based on Markov chains with more than two states are examined, followed by a discussion of the notion of absorbing Markov chains. Also covered in detail are topics relating to the average time spent in a state, various chain configurations, and n-state Markov chain simulations used for verifying experiments involving various diagram configurations. • Fascinating historical notes shed light on the key ideas that led to the development of the Markov model and its variants • Various configurations of Markov Chains and their limitations are explored at length • Numerous examples—from basic to complex—are presented in a comparative manner using a variety of color graphics • All algorithms presented can be analyzed in either Visual Basic, Java Script, or PHP • Designed to be useful to professional statisticians as well as readers without extensive knowledge of probability theory Covering both the theory underlying the Markov model and an array of Markov chain implementations, within a common conceptual framework, Markov Chains: From Theory to Implementation and Experimentation is a stimulating introduction to and a valuable reference for those wishing to deepen their understanding of this extremely valuable statistical tool. Paul A. Gagniuc, PhD, is Associate Professor at Polytechnic University of Bucharest, Romania. He obtained his MS and his PhD in genetics at the University of Bucharest. Dr. Gagniuc’s work has been published in numerous high profile scientific journals, ranging from the Public Library of Science to BioMed Central and Nature journals. He is the recipient of several awards for exceptional scientific results and a highly active figure in the review process for different scientific areas.

Markov Chains are widely used as stochastic models to study a broad spectrum of system performance and dependability characteristics. This monograph is devoted to compositional specification and analysis of Markov chains. Based on principles known from process algebra, the author systematically develops an algebra of interactive Markov chains. By presenting a number of distinguishing results, of both theoretical and practical nature, the author substantiates the claim that interactive Markov chains are more than just another formalism: Among other, an algebraic theory of interactive Markov chains is developed, devise algorithms to mechanize compositional aggregation are presented, and state spaces of several million states resulting from the study of an ordinary telefone system are analyzed.

Markov chains are a fundamental class of stochastic processes. They are widely used to solve problems in a large number of domains such as operational research, computer science, communication networks and manufacturing systems. The success of Markov chains is mainly due to their simplicity of use, the large number of available theoretical results and the quality of algorithms developed for the numerical evaluation of many metrics of interest. The author presents the theory of both discrete-time and continuous-time homogeneous Markov chains. He carefully examines the explosion phenomenon, the Kolmogorov equations, the convergence to equilibrium and the passage time distributions to a state and to a subset of states. These results are applied to birth-and-death processes. He then proposes a detailed study of the uniformization technique by means of Banach algebra. This technique is used for the transient analysis of several queuing systems. Contents 1. Discrete-Time Markov Chains 2. Continuous-Time Markov Chains 3. Birth-and-Death Processes 4. Uniformization 5. Queues About the Authors Bruno Sericola is a Senior Research Scientist at Inria Rennes – Bretagne Atlantique in France. His main research activity is in performance evaluation of computer and communication systems, dependability analysis of fault-tolerant systems and stochastic models.

This book constitutes the joint refereed proceedings of the First International Workshop on Theory Building and Formal Methods in Electronic/Mobile Commerce, TheFormEMC, the first European Performance Engineering Workshop, EPEW, and the First International Workshop on Integration of Testing Methodologies, ITM, held jointly in association with FORTE 2004 in Toledo, Spain, in October 2004. The 27 revised full papers presented were carefully reviewed and selected from a total of 62 submissions. The papers are grouped in three topical sections corresponding to the workshop topics.

This self-contained text develops a Markov chain approach that makes the rigorous analysis of a class of microscopic models that specify the dynamics of complex systems at the individual level possible. It presents a general framework of aggregation in agent-based and related computational models, one which makes use of lumpability and information theory in order to link the micro and macro levels of observation. The starting point is a microscopic Markov chain description of the dynamical process in complete correspondence with the dynamical behavior of the agent-based model (ABM), which is obtained by considering the set of all possible agent configurations as the state space of a huge Markov chain. An explicit formal representation of a resulting “micro-chain” including microscopic transition rates is derived for a class of models by using the random mapping representation of a Markov process. The type of probability distribution used to implement the stochastic part of the model, which defines the updating rule and governs the dynamics at a Markovian level, plays a crucial part in the analysis of “voter-like” models used in population genetics, evolutionary game theory and social dynamics. The book demonstrates that the problem of aggregation in ABMs - and the lumpability conditions in particular - can be embedded into a more general framework that employs information theory in order to identify different levels and relevant scales in complex dynamical systems

This volume is based on lectures given at the NATO Advanced Study Institute on "Stochastic Games and Applications," which took place at Stony Brook, NY, USA, July 1999. It gives the editors great pleasure to present it on the occasion of L.S. Shapley's eightieth birthday, and on the fiftieth "birthday" of his seminal paper "Stochastic Games," with which this volume opens. We wish to thank NATO for the grant that made the Institute and this volume possible, and the Center for Game Theory in Economics of the State University of New York at Stony Brook for hosting this event. We also wish to thank the Hebrew University of Jerusalem, Israel, for providing continuing financial support, without which this project would never have been completed. In particular, we are grateful to our editorial assistant Mike Borns, whose work has been indispensable. We also would like to acknowledge the support of the Ecole Poly tech nique, Paris, and the Israel Science Foundation. March 2003 Abraham Neyman and Sylvain Sorin ix STOCHASTIC GAMES L.S. SHAPLEY University of California at Los Angeles Los Angeles, USA 1. Introduction In a stochastic game the play proceeds by steps from position to position, according to transition probabilities controlled jointly by the two players.