Stochastic Control

Controlled Diffusions

Consider a controlled diffusion process

$dx(t) = \mu_a(x(t),t)\,dt + \sigma_a(x(t),t)\,dW(t)$

where the $μ = μ a,σ = σ a$ are parameterized by the control parameter a.

Here the objective is to minimized the expected cost, which is a combination of the running cost, and a terminal cost

$J[x(\cdot)] = \int_0^T f(x(s)) ds + g(x(T))$

Then the value function is the expected value, starting from (x,t) of the cost, assuming the optimal choice of controls is used

$u(x,t) = \min_a E^{x,t} [ J(x(\cdot)]$

The value function satisfied the Hamilton-Jacobi-Bellman equation

$-u_t = \min_a L_a u \qquad \text{ for } \qquad t < T$ ,

u (x, T) = g (x)

where $L a$ are the corresponding operators as in the linear case

$L_a u \equiv \mu_a(x,t)\frac{\partial}{\partial x}u(x,t) + \frac{1}{2}\sigma_a^2(x,t)\frac{\partial^2}{\partial x^2}u(x,t)$

The HJB equation can be regarded at the stochastic control version of the Kolmogorov backward equation.

Derivation of HJB equation

Uses

Ito
Dynamic Programming Principle
Taylor Series

Refer to Fleming Soner Controlled Markov processes and viscosity solutions

Least time problem

The least time problem is the expected time to exit a domain. Here we have zero Dirichlet boundary conditions on the inside of a bounded domain, and a running cost of 1. Instead of the fixed time T, the process runs until the stopping time, which is the first arrival on the boundary.

$\min_a L_a u(x) = 1 \qquad \text{ in } U$

$u(x) = 0 \qquad \text{ on } \partial U$

Stochastic Control - Adam Oberman Math

Contents

Introduction

Controlled Diffusions

Derivation of HJB equation

Least time problem