ray.data.datasource.BaseFileMetadataProvider

class ray.data.datasource.BaseFileMetadataProvider

Bases: ray.data.datasource.file_meta_provider.FileMetadataProvider

Abstract callable that provides metadata for FileBasedDatasource implementations that reuse the base prepare_read() method.

Also supports file and file size discovery in input directory paths.

Current subclasses:

• DefaultFileMetadataProvider

DeveloperAPI: This API may change across minor Ray releases.

Methods

__init__()
expand_paths(paths, filesystem[, ...])	Expands all paths into concrete file paths by walking directories.