jmarkov.jmdp
Class CTMDP<S extends State,A extends Action>

java.lang.Object
  extended by jmarkov.jmdp.MDP<S,A>
      extended by jmarkov.jmdp.InfiniteMDP<S,A>
          extended by jmarkov.jmdp.CTMDP<S,A>
Type Parameters:
S - The state class
A - The Action class
Direct Known Subclasses:
CTMDPEv, CTMDPEvA

public abstract class CTMDP<S extends State,A extends Action>
extends InfiniteMDP<S,A>

This class represents a continuous time MDP. It should ONLY be used in INFINITE horizon Problems. It must be extended in order to represent the appropriate structure for each INFINITE horizon MDP problem. The user must implement at least the functions that have been declared abstract. It is also necessary to create one of the extensions of the class Solver. By default, the program includes PolicyIterationSolver and ValueIterationSolver classes to solve infinite horizon problems. The FiniteSolver class is only for finite horizon problems. To solve the problem follow the instructions in each of the solvers's instructions.

Author:
Andres Sarmiento, Germán Riaño - Universidad de Los Andes
See Also:
PolicyIterationSolver, ValueIterationSolver

Field Summary
protected  S activeState
          No earthly idea what this is for..
protected  CT2DTConverter<S,A> converter
          The converter used to map the problem to a DTMDP.
protected  double maxRate
          Tha maxRate used for uniformization
 
Fields inherited from class jmarkov.jmdp.InfiniteMDP
absorbingStates, explorationTime, hasAbsorbingState, numStates, probability, probabilitySolver, states
 
Fields inherited from class jmarkov.jmdp.MDP
finite, initial, reporter
 
Constructor Summary
CTMDP(States<S> initial)
          Creates a new continuous time infinite horizon MDP Problem.
 
Method Summary
abstract  double continuousCost(S i, A a)
          Cost incurred continuously in time until the next transition from state i given that action a is taken.
protected  StatesSet<S> generate()
           
 StatesSet<S> getAllStates()
          Complete set of states explored
 double getMaxRate()
           
 ValueFunction<S> getSteadyStateProbabilities()
           
abstract  double lumpCost(S i, A a)
          Cost incurred instantaneously in the moment when action a is taken from state i.
protected  StatesSet<S> oneStageReachable(States<S> initSet)
          Finds the states reachable in one step.
abstract  double rate(S i, S j, A a)
          Rate of going from state i to state j by taking the action a
abstract  States<S> reachable(S i, A a)
          Set of States that can be reached from this state i, after taking the action a.
 void setConverter(CT2DTConverter<S,A> converter)
          Sets the class in charge of making a DTMDP equivalent to the CTMDP
 Solution<S,A> solve(double interestRate)
          Solves the problem with the given interest rate
 
Methods inherited from class jmarkov.jmdp.InfiniteMDP
feasibleActions, getDefaultAverageSolver, getDefaultDiscountedSolver, getDefaultSolver, getNumStates, getSolver, setInterestRate
 
Methods inherited from class jmarkov.jmdp.MDP
debug, debug, debug, getDebugLevel, getOptimalPolicy, getOptimalValueFunction, getReporter, isFinite, isSolved, operation, printSolution, printSolution, setDebugLevel, setReporter, setSolver, solve
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

maxRate

protected double maxRate
Tha maxRate used for uniformization


activeState

protected S extends State activeState
No earthly idea what this is for..


converter

protected CT2DTConverter<S extends State,A extends Action> converter
The converter used to map the problem to a DTMDP.

Constructor Detail

CTMDP

public CTMDP(States<S> initial)
Creates a new continuous time infinite horizon MDP Problem.

Parameters:
initial - set of initial states for the exploration algorithm
Method Detail

getAllStates

public StatesSet<S> getAllStates()
Complete set of states explored

Overrides:
getAllStates in class InfiniteMDP<S extends State,A extends Action>
Returns:
set of states explored

setConverter

public void setConverter(CT2DTConverter<S,A> converter)
Sets the class in charge of making a DTMDP equivalent to the CTMDP

Parameters:
converter - class that makes a DTMDP equivalent to the CTMDP

getMaxRate

public double getMaxRate()
Returns:
maximum exit rate for all states and all actions

oneStageReachable

protected StatesSet<S> oneStageReachable(States<S> initSet)
Finds the states reachable in one step.

Parameters:
initSet -
Returns:
States reachable from this set

generate

protected StatesSet<S> generate()
Specified by:
generate in class InfiniteMDP<S extends State,A extends Action>
Returns:
The set of states found.

getSteadyStateProbabilities

public ValueFunction<S> getSteadyStateProbabilities()
                                                           throws SolverException
Returns:
The steady state probability for each state
Throws:
SolverException

solve

public Solution<S,A> solve(double interestRate)
                                                 throws SolverException
Solves the problem with the given interest rate

Parameters:
interestRate - the interest rate parameter to solve the problem.
Returns:
The Optimal solution.
Throws:
SolverException

lumpCost

public abstract double lumpCost(S i,
                                A a)
Cost incurred instantaneously in the moment when action a is taken from state i.

Parameters:
i - State
a - Action
Returns:
Lump cost received.

continuousCost

public abstract double continuousCost(S i,
                                      A a)
Cost incurred continuously in time until the next transition from state i given that action a is taken.

Parameters:
i - State
a - Action
Returns:
Rate at which cost is incurred when action a is taken.

reachable

public abstract States<S> reachable(S i,
                                    A a)
Set of States that can be reached from this state i, after taking the action a.

Parameters:
i - current State
a - action taken
Returns:
the reachable states.

rate

public abstract double rate(S i,
                            S j,
                            A a)
Rate of going from state i to state j by taking the action a

Parameters:
i - current state
j - Destination state.
a - Action taken
Returns:
The rate