ray.train.DataConfig.configure#

DataConfig.configure(datasets: Dict[str, ray.data.dataset.Dataset], world_size: int, worker_handles: Optional[List[ray.actor.ActorHandle]], worker_node_ids: Optional[List[str]], **kwargs) → List[Dict[str, ray.data.iterator.DataIterator]][source]#

Configure how Train datasets should be assigned to workers.

Parameters

  • datasets – The datasets dict passed to Train by the user.

  • world_size – The number of Train workers in total.

  • worker_handles – The actor handles of the Train workers.

  • worker_node_ids – The node ids of the Train workers.

  • kwargs – Forwards compatibility placeholder.

Returns

A list of dataset splits for each worker. The size of the list must be equal to world_size. Each element of the list contains the assigned DataIterator instances by name for the worker.

DeveloperAPI: This API may change across minor Ray releases.