These systems play an important role in many applications.

A partially observable Markov decision process (POMDP) is a mathematical framework for planning and control problems in which actions have stochastic effects and observations provide uncertain state information. It is widely used for research in decision-theoretic planning and reinforcement learning. To cope with partial observability, a policy (or plan) must use memory, and previous work has shown that a finite-state controller provides a good policy representation. This thesis considers a previously-developed bounded policy iteration algorithm for POMDPs that finds policies that take the form of stochastic finite-state controllers. Two new improvements of this algorithm are developed. First improvement provides a simplification of the basic linear program, which is used to find improved controllers. This results in a considerable speed-up in efficiency of the original algorithm. Secondly, a branch and bound algorithm for adding the best possible node to the controller is presented, which provides an error bound and a test for global optimality. Experimental results show that these enhancements significantly improve the algorithm's performance.

This paper is concerned with the optimal control of continuous-time Markov processes. The admissible control laws are based on white-noise corrupted observations of a function on the state processes. A 'separated' control problem is introduced, whose states are probability measures on the original state space. The original and separated control problems are related via the nonlinear filter equation. The existence of a minimum for the separated problem is established. Under more restrictive assumptions it is shown that the minimum expected cost for the separated problem equals the infimum of expected costs for the original problem with partially observed states.