icenet.data.interfaces package

Submodules

icenet.data.interfaces.cds module

class icenet.data.interfaces.cds.ERA5Downloader(*args, identifier: str = 'era5', cdi_map: object = {'hus': 'specific_humidity', 'psl': 'surface_pressure', 'rlds': 'surface_thermal_radiation_downwards', 'rsds': 'surface_solar_radiation_downwards', 'ta': 'temperature', 'tas': '2m_temperature', 'tos': 'sea_surface_temperature', 'uas': '10m_u_component_of_wind', 'vas': '10m_v_component_of_wind', 'zg': 'geopotential'}, use_toolbox: bool = False, show_progress: bool = False, **kwargs)

Bases: ClimateDownloader

Climate downloader to provide ERA5 reanalysis data from CDS API

Parameters:

• identifier – how to identify this dataset

• cdi_map – override the default ERA5Downloader.CDI_MAP variable map

• use_toolbox – whether to use CDS toolbox for remote aggregation

• show_progress – whether to show download progress

CDI_MAP = {'hus': 'specific_humidity', 'psl': 'surface_pressure', 'rlds': 'surface_thermal_radiation_downwards', 'rsds': 'surface_solar_radiation_downwards', 'ta': 'temperature', 'tas': '2m_temperature', 'tos': 'sea_surface_temperature', 'uas': '10m_u_component_of_wind', 'vas': '10m_v_component_of_wind', 'zg': 'geopotential'}

additional_regrid_processing(datafile: str, cube_ease: object)

Parameters:

• datafile

• cube_ease

postprocess(var: str, download_path: object)

Processing of CDS downloaded files

If we’ve not used the toolbox to download the files, we have a lot of hourly data to average out, which is taken care of here

Parameters:

• var

• download_path

icenet.data.interfaces.cds.main()

icenet.data.interfaces.downloader module

class icenet.data.interfaces.downloader.ClimateDownloader(*args, dates: object = (), delete_tempfiles: bool = True, download: bool = True, drop_vars: list = None, group_dates_by: str = 'year', levels: object = (), max_threads: int = 1, postprocess: bool = True, pregrid_prefix: str = 'latlon_', var_name_idx: int = -1, var_names: object = (), **kwargs)

Bases: Downloader

Climate downloader base class

Parameters:

• dates

• delete_tempfiles

• download

• group_dates_by

• max_threads

• postprocess

• pregrid_prefix

• levels

• var_name_idx

• var_names

abstract additional_regrid_processing(datafile: str, cube_ease: object)

Parameters:

• datafile

• cube_ease

convert_cube(cube: object)

Converts Iris cube to be fit for regrid

Parameters:

cube – the cube requiring alteration

Return cube:

the altered cube

property dates

property delete

download()

Handles concurrent (threaded) downloading for variables

This takes dates, variables and levels as configured, batches them into requests and submits those via a ThreadPoolExecutor for concurrent downloading. Returns nothing, relies on _single_download to implement appropriate updates to this object to record state changes arising from downloading.

property download_method: callable

get_req_filenames(var_folder: str, req_date: object, date_format: str = None)

Parameters:

• var_folder

• req_date

• date_format

Returns:

property group_dates_by

property levels

postprocess(var, download_path)

property pregrid_prefix

regrid(files: object = None, rotate_wind: bool = True)

Parameters:

files

rotate_wind_data(apply_to: object = ('uas', 'vas'), manual_files: object = None)

Parameters:

• apply_to

• manual_files

save_temporal_files(var, da, date_format=None, freq=None)

Parameters:

• var

• da

• date_format

• freq

property sic_ease_cube

return sic_cube:

property var_names

icenet.data.interfaces.downloader.filter_dates_on_data(latlon_path: str, regridded_name: str, req_dates: object, check_latlon: bool = True, check_regridded: bool = True, drop_vars: list = None)

Reduces request dates and target files based on existing data

To avoid what is potentially significant resource expense downloading extant data, downloaders should call this method to reduce the request dates only to that data not already present. This is a fairly naive implementation, in that if the data is present in either the latlon intermediate file OR the target regridded file, we’ll not bother downloading again. This can be overridden via the method arguments.

Parameters:

• latlon_path

• regridded_name

• req_dates

• check_latlon

• check_regridded

• drop_vars

Returns:

req_dates(list)

icenet.data.interfaces.downloader.merge_files(new_datafile: str, other_datafile: str, drop_variables: object = None)

Parameters:

• new_datafile

• other_datafile

• drop_variables

icenet.data.interfaces.esgf module

class icenet.data.interfaces.esgf.CMIP6Downloader(*args, source: str, member: str, nodes: object = ('esgf.ceda.ac.uk', 'esg1.umr-cnrm.fr', 'vesg.ipsl.upmc.fr', 'esgf3.dkrz.de', 'esgf.bsc.es', 'esgf-data.csc.fi', 'noresg.nird.sigma2.no', 'esgf-data.ucar.edu', 'esgf-data2.diasjp.net'), experiments: object = ('historical', 'ssp245'), frequency: str = 'day', table_map: object = None, grid_map: object = None, grid_override: object = None, exclude_nodes: object = None, **kwargs)

Bases: ClimateDownloader

Climate downloader to provide CMIP6 reanalysis data from ESGF APIs

Useful CMIP6 guidance: https://pcmdi.llnl.gov/CMIP6/Guide/dataUsers.html

Parameters:

• identifier – how to identify this dataset

• source – source ID in ESGF node

• member – member ID in ESGF node

• nodes – list of ESGF nodes to query

• experiments – experiment IDs to download

• frequency – query parameter frequency

• table_map – table map for

• grid_map

• grid_override

• exclude_nodes

“MRI-ESM2-0”, “r1i1p1f1”, None “MRI-ESM2-0”, “r2i1p1f1”, None “MRI-ESM2-0”, “r3i1p1f1”, None “MRI-ESM2-0”, “r4i1p1f1”, None “MRI-ESM2-0”, “r5i1p1f1”, None “EC-Earth3”, “r2i1p1f1”, “gr” “EC-Earth3”, “r7i1p1f1”, “gr” “EC-Earth3”, “r10i1p1f1”, “gr” “EC-Earth3”, “r12i1p1f1”, “gr” “EC-Earth3”, “r14i1p1f1”, “gr”

ESGF_NODES = ('esgf.ceda.ac.uk', 'esg1.umr-cnrm.fr', 'vesg.ipsl.upmc.fr', 'esgf3.dkrz.de', 'esgf.bsc.es', 'esgf-data.csc.fi', 'noresg.nird.sigma2.no', 'esgf-data.ucar.edu', 'esgf-data2.diasjp.net')

GRID_MAP = {'hus': 'gn', 'psl': 'gn', 'rlds': 'gn', 'rsds': 'gn', 'rsus': 'gn', 'siconca': 'gn', 'ta': 'gn', 'tas': 'gn', 'tos': 'gr', 'ua': 'gn', 'uas': 'gn', 'vas': 'gn', 'zg': 'gn'}

TABLE_MAP = {'hus': 'day', 'psl': 'day', 'rlds': 'day', 'rsds': 'day', 'rsus': 'day', 'siconca': 'SIday', 'ta': 'day', 'tas': 'day', 'tos': 'Oday', 'ua': 'day', 'uas': 'day', 'vas': 'day', 'zg': 'day'}

additional_regrid_processing(datafile: str, cube_ease: object)

Parameters:

• datafile

• cube_ease

convert_cube(cube: object) → object

Converts Iris cube to be fit for CMIP regrid

Parameters:

cube – the cube requiring alteration

Return cube:

the altered cube

icenet.data.interfaces.esgf.main()

icenet.data.interfaces.mars module

class icenet.data.interfaces.mars.HRESDownloader(*args, identifier: str = 'mars.hres', **kwargs)

Bases: ClimateDownloader

Climate downloader to provide CMIP6 reanalysis data from ESGF APIs

Parameters:

identifier – how to identify this dataset

HRES_PARAMS = {'hus': (133, 'q'), 'psl': (134, 'sp'), 'rlds': (175, 'strd'), 'rsds': (169, 'ssrd'), 'siconca': (31, 'siconc'), 'ta': (130, 't'), 'tas': (167, 't2m'), 'tos': (34, 'sst'), 'uas': (165, 'u10'), 'vas': (166, 'v10'), 'zg': (129, 'z')}

MARS_TEMPLATE = '\nretrieve,\n  class=od,\n  date={date},\n  expver=1,\n  levtype={levtype},\n  {levlist}param={params},\n  step={step},\n  stream=oper,\n  time=12:00:00,\n  type=fc,\n  area={area},\n  grid=0.25/0.25,\n  target="{target}",\n  format=netcdf\n    '

PARAM_TABLE = 128

additional_regrid_processing(datafile: str, cube_ease: object)

Parameters:

• datafile

• cube_ease

download()

property mars_template

property param_table

property params

class icenet.data.interfaces.mars.SEASDownloader(*args, identifier: str = 'mars.hres', **kwargs)

Bases: HRESDownloader

MARS_TEMPLATE = '\nretrieve,\n    class=od,\n    date={date},\n    expver=1,\n    levtype={levtype},\n    method=1,\n    number=0/1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24,\n    origin=ecmf,\n    {levlist}param={params},\n    step=0/to/2232/by/24,\n    stream=mmsf,\n    system=5,\n    time=00:00:00,\n    type=fc,\n    target="{target}",\n    format=netcdf,\n    grid=0.25/0.25,\n    area={area}'

save_temporal_files(var, da, date_format=None, freq=None)

Parameters:

• var

• da

• date_format

• freq

icenet.data.interfaces.mars.hres_main()

icenet.data.interfaces.mars.main(identifier, extra_kwargs=None)

icenet.data.interfaces.mars.seas_main()

icenet.data.interfaces.utils module

icenet.data.interfaces.utils.add_time_dim(source: str, hemisphere: str, identifier: str, dry: bool = False, var_names: object = [])

Parameters:

• source

• hemisphere

• identifier

• dry

• var_names

icenet.data.interfaces.utils.add_time_dim_main()

CLI stub to sort out missing time dimensions in daily data

icenet.data.interfaces.utils.batch_requested_dates(dates: object, attribute: str = 'month') → object

TODO: should be using Pandas DatetimeIndexes / Periods for this, but the

need to refactor slightly, and this is working for the moment

Parameters:

• dates

• attribute

Returns:

icenet.data.interfaces.utils.get_args()

Returns:

icenet.data.interfaces.utils.reprocess_main()

CLI stub solution for reprocessing monthly files

icenet.data.interfaces.utils.reprocess_monthlies(source: str, hemisphere: str, identifier: str, output_base: str, dry: bool = False, var_names: object = None)

Parameters:

• source

• hemisphere

• identifier

• output_base

• dry

• var_names

Module contents