jmdp
Class CTMDP<S extends State,A extends Action>

java.lang.Object
  extended by jmdp.MDP<S,A>
      extended by jmdp.InfiniteMDP<S,A>
          extended by jmdp.CTMDP<S,A>
Direct Known Subclasses:
CTMDPEv, CTMDPEvA

public abstract class CTMDP<S extends State,A extends Action>
extends InfiniteMDP<S,A>

This class represents a continuous time MDP. It should ONLY be used in INFINITE Problems. It must be extended in order to represent the appropriate structure for each INFINITE horizon MDP problem. The user must implement at least the functions that have been declared abstract. It is also necessary to create one of the extensions of the class Solver. By default, the program includes PolicyIterationSolver and ValueIterationSolver classes to solve infinite horizon problems. The FiniteSolver class is only for finite horizon problems. To solve the problem follow the instructions in each of the solvers's instructions.

Author:
Andres Sarmiento, German Riano - Universidad de Los Andes
See Also:
PolicyIterationSolver, ValueIterationSolver

Field Summary
protected  S activeState
           
protected  java.util.TreeMap<A,java.lang.Double> exitRates
           
protected  double maxRate
           
 
Fields inherited from class jmdp.InfiniteMDP
absorbingStates, explorationTime, hasAbsorbingState, probability, probabilitySolver, states
 
Fields inherited from class jmdp.MDP
finite, initial, reporter
 
Constructor Summary
CTMDP(States<S> initial)
          Creates a new continuous time infinite horizon MDP Problem.
 
Method Summary
abstract  double continuousCost(S i, A a)
          Reward obtained continuously in time until the next transition from state i given that action a is taken.
protected  double exitRate(S i, A a)
          Exit rate from state i given action a is taken
protected  States<S> explore(States<S> initSet)
           
 States<S> getAllStates()
          Complete set of states explored
 double getMaxRate()
           
 ValueFunction<S> getSteadyStateProbabilities()
           
abstract  double lumpCost(S i, A a)
          Reward instantaneously in the moment when action a is taken from state i.
protected  States<S> oneStageReachable(States<S> initSet)
           
abstract  double rate(S i, S j, A a)
          Rate of going from state i to state j by taking the action a
abstract  States<S> reached(S i, A a)
          Set of states that can be reached from this state i, after taking the action a.
 void setConverter(CT2DTConverter<S,A> converter)
          Sets the class in charge of making a DTMDP equivalent to the CTMDP
 Solution<S,A> solve(double interestRate)
          Solves the problem with the given interest rate
 
Methods inherited from class jmdp.InfiniteMDP
feasibleActions, getDefaultAverageSolver, getDefaultDiscountedSolver, getDefaultSolver, getSolver, setInterestRate
 
Methods inherited from class jmdp.MDP
debug, getOptimalPolicy, getOptimalValueFunction, getReporter, isFinite, isSolved, operation, printSolution, printSolution, setReporter, setSolver, solve
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

maxRate

protected double maxRate

exitRates

protected java.util.TreeMap<A extends Action,java.lang.Double> exitRates

activeState

protected S extends State activeState
Constructor Detail

CTMDP

public CTMDP(States<S> initial)
Creates a new continuous time infinite horizon MDP Problem.

Parameters:
initial - set of initial states for the exploration algorithm
Method Detail

getAllStates

public States<S> getAllStates()
Complete set of states explored

Overrides:
getAllStates in class InfiniteMDP<S extends State,A extends Action>
Returns:
set of states explored

setConverter

public void setConverter(CT2DTConverter<S,A> converter)
Sets the class in charge of making a DTMDP equivalent to the CTMDP

Parameters:
converter - class that makes a DTMDP equivalent to the CTMDP

getMaxRate

public double getMaxRate()
Returns:
maximum exit rate for all states and all actions

exitRate

protected double exitRate(S i,
                          A a)
Exit rate from state i given action a is taken

Parameters:
i - current state
a - action taken
Returns:
exit rate

oneStageReachable

protected States<S> oneStageReachable(States<S> initSet)

explore

protected States<S> explore(States<S> initSet)
Specified by:
explore in class InfiniteMDP<S extends State,A extends Action>

getSteadyStateProbabilities

public ValueFunction<S> getSteadyStateProbabilities()
Returns:
a Map with the steady state probability for each state

solve

public Solution<S,A> solve(double interestRate)
Solves the problem with the given interest rate

Parameters:
interestRate - the interest rate parameter to solve the problem.

lumpCost

public abstract double lumpCost(S i,
                                A a)
Reward instantaneously in the moment when action a is taken from state i.


continuousCost

public abstract double continuousCost(S i,
                                      A a)
Reward obtained continuously in time until the next transition from state i given that action a is taken.


reached

public abstract States<S> reached(S i,
                                  A a)
Set of states that can be reached from this state i, after taking the action a.


rate

public abstract double rate(S i,
                            S j,
                            A a)
Rate of going from state i to state j by taking the action a