icenet.data.loaders package

Submodules

icenet.data.loaders.base module

class icenet.data.loaders.base.IceNetBaseDataLoader(configuration_path: str, identifier: str, var_lag: int, *args, dataset_config_path: str = '.', dates_override: object = None, dry: bool = False, generate_workers: int = 8, loss_weight_days: bool = True, n_forecast_days: int = 93, output_batch_size: int = 32, path: str = './network_datasets', pickup: bool = False, var_lag_override: object = None, **kwargs)[source]

Bases: Generator

:param configuration_path, :param identifier, :param var_lag, :param dataset_config_path: :param generate_workers: :param loss_weight_days: :param n_forecast_days: :param output_batch_size: :param path: :param var_lag_override:

property channel_names
property config
property dates_override
abstract generate_sample(date: object, prediction: bool = False)[source]
Parameters:

  • date

  • prediction

Returns:

get_sample_files() → object[source]
Parameters:

date

Returns:

property num_channels
property pickup
property workers
write_dataset_config_only()[source]

icenet.data.loaders.dask module

class icenet.data.loaders.dask.DaskBaseDataLoader(*args, dask_port: int = 8888, dask_timeouts: int = 60, dask_tmp_dir: object = '/tmp', **kwargs)[source]

Bases: IceNetBaseDataLoader

A subclass of IceNetBaseDataLoader that provides functionality for loading data using Dask.

_dashboard_port

The port number for the Dask dashboard.

_timeout

The timeout value for Dask communication.

_tmp_dir

The temporary directory for Dask.

client_generate(client: object, dates_override: object = None, pickup: bool = False) → None[source]

Generates data using the Dask client. This method needs to be implemented in subclasses.

Parameters:

  • client – The Dask client.

  • dates_override (optional) – A dict with keys train, val, test, each with a list of continuous dates for that purpose. Defaults to None.

  • pickup (optional) – TODO. Defaults to False.

Raises:

NotImplementedError – If generate is called without being implemented as a subclass of DaskBaseDataLoader.

generate() → None[source]

Generates data using Dask client by setting up a Dask cluster and client, and calling client_generate method.

class icenet.data.loaders.dask.DaskMultiSharingWorkerLoader(*args, **kwargs)[source]

Bases: DaskBaseDataLoader

client_generate(client: object, dates_override: object = None, pickup: bool = False)[source]
Parameters:

  • client

  • dates_override

  • pickup

generate_sample(date: object, prediction: bool = False)[source]
Parameters:

  • date

  • prediction

class icenet.data.loaders.dask.DaskMultiWorkerLoader(*args, futures_per_worker: int = 2, **kwargs)[source]

Bases: DaskBaseDataLoader

client_generate(client: object, dates_override: object = None, pickup: bool = False)[source]
Parameters:

  • client

  • dates_override

  • pickup

generate_sample(date: object, prediction: bool = False, parallel=True)[source]
Parameters:

  • date

  • prediction

Returns:

icenet.data.loaders.dask.generate_and_write(path: str, var_files: object, dates: object, args: tuple, dry: bool = False)[source]
Parameters:

  • path

  • var_files

  • dates

  • args

  • dry

Returns:

icenet.data.loaders.dask.generate_sample(forecast_date: object, var_ds: object, var_files: object, trend_ds: object, channels: object, dtype: object, loss_weight_days: bool, meta_channels: object, missing_dates: object, n_forecast_days: int, num_channels: int, shape: object, trend_steps: object, masks: object, prediction: bool = False)[source]
Parameters:

  • forecast_date

  • var_ds

  • var_files

  • trend_ds

  • channels

  • dtype

  • loss_weight_days

  • meta_channels

  • missing_dates

  • n_forecast_days

  • num_channels

  • shape

  • trend_steps

  • masks

  • prediction

Returns:

icenet.data.loaders.stdlib module

class icenet.data.loaders.stdlib.IceNetDataLoader(*args, **kwargs)[source]

Bases: IceNetBaseDataLoader

generate()[source]
generate_sample(date: object, prediction: bool = False)[source]
Parameters:

  • date

  • prediction

icenet.data.loaders.utils module

exception icenet.data.loaders.utils.IceNetDataWarning[source]

Bases: RuntimeWarning

icenet.data.loaders.utils.write_tfrecord(writer: object, x: object, y: object, sample_weights: object)[source]
Parameters:

  • writer

  • x

  • y

  • sample_weights

  • data_check

Module contents

class icenet.data.loaders.IceNetDataLoaderFactory[source]

Bases: object

A factory class for managing a map of loader names and their corresponding implementation classes.

_loader_map

A dictionary holding loader names against their implementation classes.

add_data_loader(loader_name: str, loader_impl: object) → None[source]

Adds a new loader to the loader map with the given name and implementation class.

Parameters:

  • loader_name – The name of the loader.

  • loader_impl – The implementation class of the loader.

Returns:

None. Updates _loader_map attribute in IceNetDataLoaderFactory with specified

loader name and implementation.

Raises:

RuntimeError – If the loader name already exists or if the implementation class is not a descendant of IceNetBaseDataLoader.

create_data_loader(loader_name, *args, **kwargs) → object[source]

Creates an instance of a loader based on specified name from the _loader_map dict attribute.

Parameters:

  • loader_name – The name of the loader.

  • *args – Additional positional arguments, is passed to the loader constructor.

  • **kwargs – Additional keyword arguments, is passed to the loader constructor.

Returns:

An instance of the loader class.

Raises:

KeyError – If the loader name does not exist in _loader_map.

property loader_map: dict

The loader map dictionary.