Sampling the Environment or offline data
Contents
Sampling the Environment or offline data#
Data ingest via either environment rollouts or other data-generating methods
(e.g. reading from offline files) is done in RLlib by RolloutWorker instances,
which sit inside a WorkerSet
(together with other parallel RolloutWorkers) in the RLlib Algorithm
(under the self.workers property):
A typical RLlib WorkerSet setup inside an RLlib Algorithm: Each WorkerSet contains
exactly one local RolloutWorker object and N ray remote
RolloutWorker (ray actors).
The workers contain a policy map (with one or more policies), and - in case a simulator
(env) is available - a vectorized BaseEnv
(containing M sub-environments) and a SamplerInput (either synchronous or asynchronous) which controls
the environment data collection loop.
In the online case (i.e. environment is available) as well as the offline case (i.e. no environment),
Algorithm uses the sample() method to
get SampleBatch objects for training.#
RolloutWorker API#
Constructor#
Multi agent#
Setter and getter methods#
Threading#
Sampling API#
Training API#
Environment API#
Miscellaneous#
WorkerSet API#
Constructor#
Worker Orchestration#
Pass-through methods#
Sampler API#
InputReader instances are used to collect and return experiences from the envs.
For more details on InputReader used for offline RL (e.g. reading files of
pre-recorded data), see the offline RL API reference here.
Input Reader API#
Input Sampler API#
Synchronous Sampler API#
Asynchronous Sampler API#
Offline Sampler API#
The InputReader API is used by an individual RolloutWorker
to produce batches of experiences either from an simulator or from an
offline source (e.g. a file).
Here are some example extentions of the InputReader API: