Accession Number : ADA563366


Title :   Automatic State Space Aggregation Using a Density Based Technique


Descriptive Note : Conference paper


Corporate Author : AIR FORCE RESEARCH LAB ROME NY INFORMATION DIRECTORATE


Personal Author(s) : Wright, Robert ; Loscalzo, Steven


Full Text : https://apps.dtic.mil/dtic/tr/fulltext/u2/a563366.pdf


Report Date : May 2012


Pagination or Media Count : 9


Abstract : applying reinforcement learning techniques in continuous environments is challenging because there are infinitely many states to visit in order to learn an optimal policy. To make this situation tractable, abstractions are often used to reduce the infinite state space down to a small and finite one. Some of the more powerful and commonplace abstractions, tiling abstractions such as CMAC, work by aggregating many base states into a single abstract state. Unfortunately, significant manual effort is often necessary in order to apply them to non-trivial control problems. Here we develop an automatic state space aggregation algorithm, Maximum Density Separation, which can produce a meaningful abstraction with minimal manual effort. This method leverages the density of observations in the space to construct a partition and aggregate states in a dense region to the same abstract state. We show that the abstractions produced by this method on two benchmark reinforcement learning problems can outperform fixed tiling methods in terms of both the convergence rate of a learning algorithm and the number of abstract states needed.


Descriptors :   *LEARNING MACHINES , ALGORITHMS , ARTIFICIAL INTELLIGENCE , AUTOMATIC , SYMPOSIA


Subject Categories : Operations Research
      Cybernetics


Distribution Statement : APPROVED FOR PUBLIC RELEASE