3: IceNet Data and Forecast Products

3: IceNet Data and Forecast Products#

Context#

Purpose#

The IceNet library provides the ability to download, process, train and predict from end to end via a set of command-line interfaces (CLI).

Using this notebook one can understand the various data sources, intermediaries and products that arise from both the CLI demonstrator notebook and Pipeline demonstrate notebook activities.

Modelling approach#

This modelling approach allows users to immediately utilise the library for producing sea ice concentraion forecasts.

Highlights#

The key features of an end to end run are:

Setup: this was concerned with setting up the conda environment, which remains the same
1. Download: we explore the source data downloaded under the /data/ folder, reusable across multiple environments
2. Process: we explore the preprocessing outputs in the /processed/ folder, which can be fed to models directly or further processed into IceNet datasets
3. Train: we explore the outputs from the training process and ensemble runs, stored within /results/networks/
4. Predict: we explore the output from the prediction process and ensemble runs, stored within /results/predict/

This follows the same structure as the CLI demonstration notebook so that it’s easy to follow step-by-step…

Contributions#

Notebook#

James Byrne (author)

Bryn Noel Ubald (co-author)

Please raise issues in this repository to suggest updates to this notebook!

Contact me at jambyr <at> bas.ac.uk for anything else…

Modelling codebase#

James Byrne (code author), Bryn Noel Ubald (code author), Tom Andersson (science author)

Modelling publications#

Andersson, T.R., Hosking, J.S., Pérez-Ortiz, M. et al. Seasonal Arctic sea ice forecasting with probabilistic deep learning. Nat Commun 12, 5124 (2021). https://doi.org/10.1038/s41467-021-25257-4

Involved organisations#

The Alan Turing Institute and British Antarctic Survey

Setup#

For the purposes of python analysis we use and provide the following header libraries which are heavily utilised within the IceNet project and the pipeline.

import glob, json, os
import datetime as dt
import pandas as pd, xarray as xr, matplotlib.pyplot as plt
from IPython.display import HTML
%matplotlib inline

1. Download#

Downloading data using the icenet_data commands produces a dataset specific input data storage directory called /data whose source data can be reused across normalisation (icenet_process*) and dataset production (icenet_dataset*) runs.

os.listdir("data")

['masks', 'era5', 'osisaf']

The structure of these directories (aside from masks) have consistent layouts:

glob.glob("data/**/2020.nc", recursive=True)

['data/era5/north/psl/2020.nc',
 'data/era5/north/ta500/2020.nc',
 'data/era5/north/tas/2020.nc',
 'data/era5/north/tos/2020.nc',
 'data/era5/north/uas/2020.nc',
 'data/era5/north/vas/2020.nc',
 'data/era5/north/zg250/2020.nc',
 'data/era5/north/zg500/2020.nc',
 'data/era5/south/psl/2020.nc',
 'data/era5/south/ta500/2020.nc',
 'data/era5/south/tas/2020.nc',
 'data/era5/south/tos/2020.nc',
 'data/era5/south/uas/2020.nc',
 'data/era5/south/vas/2020.nc',
 'data/era5/south/zg250/2020.nc',
 'data/era5/south/zg500/2020.nc',
 'data/osisaf/south/siconca/2020.nc']

With masks being the only caveat. However, masks can be interacted with purely through the Masks class from icenet.data.sic.mask.

glob.glob("data/masks/**/*.*", recursive=True)

['data/masks/south/masks/masks.params',
 'data/masks/south/masks/active_grid_cell_mask_01.npy',
 'data/masks/south/masks/land_mask.npy',
 'data/masks/south/masks/active_grid_cell_mask_02.npy',
 'data/masks/south/masks/active_grid_cell_mask_03.npy',
 'data/masks/south/masks/active_grid_cell_mask_04.npy',
 'data/masks/south/masks/active_grid_cell_mask_05.npy',
 'data/masks/south/masks/active_grid_cell_mask_06.npy',
 'data/masks/south/masks/active_grid_cell_mask_07.npy',
 'data/masks/south/masks/active_grid_cell_mask_08.npy',
 'data/masks/south/masks/active_grid_cell_mask_09.npy',
 'data/masks/south/masks/active_grid_cell_mask_10.npy',
 'data/masks/south/masks/active_grid_cell_mask_11.npy',
 'data/masks/south/masks/active_grid_cell_mask_12.npy',
 'data/masks/south/siconca/2000/01/ice_conc_sh_ease2-250_cdr-v2p0_200001021200.nc',
 'data/masks/south/siconca/2000/02/ice_conc_sh_ease2-250_cdr-v2p0_200002021200.nc',
 'data/masks/south/siconca/2000/03/ice_conc_sh_ease2-250_cdr-v2p0_200003021200.nc',
 'data/masks/south/siconca/2000/04/ice_conc_sh_ease2-250_cdr-v2p0_200004021200.nc',
 'data/masks/south/siconca/2000/05/ice_conc_sh_ease2-250_cdr-v2p0_200005021200.nc',
 'data/masks/south/siconca/2000/06/ice_conc_sh_ease2-250_cdr-v2p0_200006021200.nc',
 'data/masks/south/siconca/2000/07/ice_conc_sh_ease2-250_cdr-v2p0_200007021200.nc',
 'data/masks/south/siconca/2000/08/ice_conc_sh_ease2-250_cdr-v2p0_200008021200.nc',
 'data/masks/south/siconca/2000/09/ice_conc_sh_ease2-250_cdr-v2p0_200009021200.nc',
 'data/masks/south/siconca/2000/10/ice_conc_sh_ease2-250_cdr-v2p0_200010021200.nc',
 'data/masks/south/siconca/2000/11/ice_conc_sh_ease2-250_cdr-v2p0_200011021200.nc',
 'data/masks/south/siconca/2000/12/ice_conc_sh_ease2-250_cdr-v2p0_200012021200.nc']

Note that the siconca variable files are source files for generating the masks that are not actually used after initial mask creation.

Producing data source videos#

One of the easiest ways to inspect source data is to use the icenet_video_data command, which will output to /plot/ in the run directory video(s) corresponding to the selected dataset, hemisphere, variable and year, or all of these items if run with:

!icenet_video_data --years 2020 era5,osisaf

[18-02-25 17:22:14 :INFO    ] - Looking into data
[18-02-25 17:22:14 :INFO    ] - Looking at data
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/north
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/north/psl
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/north/siconca
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/north/ta500
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/north/tas
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/north/tos
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/north/uas
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/north/vas
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/north/zg250
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/north/zg500
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/south
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/south/psl
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/south/siconca
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/south/ta500
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/south/tas
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/south/tos
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/south/uas
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/south/vas
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/south/zg250
[18-02-25 17:22:14 :INFO    ] - Looking at data/era5/south/zg500
[18-02-25 17:22:14 :INFO    ] - Looking at data/osisaf
[18-02-25 17:22:14 :INFO    ] - Looking at data/osisaf/south
[18-02-25 17:22:14 :INFO    ] - Looking at data/osisaf/south/siconca
[18-02-25 17:22:14 :INFO    ] - Saving to plot/data_era5_north_ta500.mp4
[18-02-25 17:22:14 :INFO    ] - Saving to plot/data_era5_north_psl.mp4
[18-02-25 17:22:14 :INFO    ] - Saving to plot/data_era5_north_uas.mp4
[18-02-25 17:22:14 :INFO    ] - Saving to plot/data_era5_north_tas.mp4
[18-02-25 17:22:14 :INFO    ] - Saving to plot/data_era5_north_vas.mp4
[18-02-25 17:22:14 :INFO    ] - Saving to plot/data_era5_north_zg250.mp4
[18-02-25 17:22:14 :INFO    ] - Saving to plot/data_era5_north_zg500.mp4
[18-02-25 17:22:14 :INFO    ] - Saving to plot/data_era5_north_tos.mp4
[18-02-25 17:22:14 :INFO    ] - Inspecting data
[18-02-25 17:22:14 :INFO    ] - Inspecting data
[18-02-25 17:22:14 :INFO    ] - Inspecting data
[18-02-25 17:22:14 :INFO    ] - Inspecting data
[18-02-25 17:22:14 :INFO    ] - Inspecting data
[18-02-25 17:22:14 :INFO    ] - Inspecting data
[18-02-25 17:22:14 :INFO    ] - Inspecting data
[18-02-25 17:22:14 :INFO    ] - Inspecting data
[18-02-25 17:22:15 :INFO    ] - Initialising plot
[18-02-25 17:22:15 :INFO    ] - Initialising plot
[18-02-25 17:22:15 :INFO    ] - Initialising plot
[18-02-25 17:22:15 :INFO    ] - Initialising plot
[18-02-25 17:22:15 :INFO    ] - Initialising plot
[18-02-25 17:22:15 :INFO    ] - Initialising plot
[18-02-25 17:22:15 :INFO    ] - Initialising plot
[18-02-25 17:22:16 :INFO    ] - Initialising plot
[18-02-25 17:22:16 :INFO    ] - Animating
[18-02-25 17:22:16 :INFO    ] - Animating
[18-02-25 17:22:17 :INFO    ] - Animating
[18-02-25 17:22:17 :INFO    ] - Animating
[18-02-25 17:22:17 :INFO    ] - Animating
[18-02-25 17:22:17 :INFO    ] - Animating
[18-02-25 17:22:17 :INFO    ] - Animating
[18-02-25 17:22:17 :INFO    ] - Animating
[18-02-25 17:22:17 :INFO    ] - Saving plot to plot/data_era5_north_psl.mp4
[18-02-25 17:22:17 :INFO    ] - Saving plot to plot/data_era5_north_zg500.mp4
[18-02-25 17:22:17 :INFO    ] - Saving plot to plot/data_era5_north_tas.mp4
[18-02-25 17:22:17 :INFO    ] - Saving plot to plot/data_era5_north_tos.mp4
[18-02-25 17:22:17 :INFO    ] - Saving plot to plot/data_era5_north_ta500.mp4
[18-02-25 17:22:17 :INFO    ] - Saving plot to plot/data_era5_north_zg250.mp4
[18-02-25 17:22:18 :INFO    ] - Saving plot to plot/data_era5_north_uas.mp4
[18-02-25 17:22:18 :INFO    ] - Saving plot to plot/data_era5_north_vas.mp4
[18-02-25 17:25:07 :INFO    ] - Produced plot/data_era5_north_tos.mp4
[18-02-25 17:25:07 :INFO    ] - Saving to plot/data_era5_south_psl.mp4
[18-02-25 17:25:07 :INFO    ] - Inspecting data
[18-02-25 17:25:08 :INFO    ] - Initialising plot
[18-02-25 17:25:08 :INFO    ] - Animating
[18-02-25 17:25:08 :INFO    ] - Saving plot to plot/data_era5_south_psl.mp4
[18-02-25 17:25:23 :INFO    ] - Produced plot/data_era5_north_tas.mp4
[18-02-25 17:25:23 :INFO    ] - Saving to plot/data_era5_south_ta500.mp4
[18-02-25 17:25:23 :INFO    ] - Inspecting data
[18-02-25 17:25:24 :INFO    ] - Initialising plot
[18-02-25 17:25:24 :INFO    ] - Animating
[18-02-25 17:25:24 :INFO    ] - Saving plot to plot/data_era5_south_ta500.mp4
[18-02-25 17:25:25 :INFO    ] - Produced plot/data_era5_north_psl.mp4
[18-02-25 17:25:25 :INFO    ] - Saving to plot/data_era5_south_tas.mp4
[18-02-25 17:25:25 :INFO    ] - Inspecting data
[18-02-25 17:25:26 :INFO    ] - Initialising plot
[18-02-25 17:25:26 :INFO    ] - Animating
[18-02-25 17:25:26 :INFO    ] - Produced plot/data_era5_north_zg500.mp4
[18-02-25 17:25:26 :INFO    ] - Saving to plot/data_era5_south_tos.mp4
[18-02-25 17:25:26 :INFO    ] - Inspecting data
[18-02-25 17:25:27 :INFO    ] - Saving plot to plot/data_era5_south_tas.mp4
[18-02-25 17:25:27 :INFO    ] - Initialising plot
[18-02-25 17:25:27 :INFO    ] - Animating
[18-02-25 17:25:28 :INFO    ] - Saving plot to plot/data_era5_south_tos.mp4
[18-02-25 17:25:30 :INFO    ] - Produced plot/data_era5_north_ta500.mp4
[18-02-25 17:25:30 :INFO    ] - Saving to plot/data_era5_south_uas.mp4
[18-02-25 17:25:30 :INFO    ] - Inspecting data
[18-02-25 17:25:31 :INFO    ] - Initialising plot
[18-02-25 17:25:32 :INFO    ] - Animating
[18-02-25 17:25:33 :INFO    ] - Saving plot to plot/data_era5_south_uas.mp4
[18-02-25 17:25:33 :INFO    ] - Produced plot/data_era5_north_zg250.mp4
[18-02-25 17:25:33 :INFO    ] - Saving to plot/data_era5_south_vas.mp4
[18-02-25 17:25:33 :INFO    ] - Inspecting data
[18-02-25 17:25:34 :INFO    ] - Initialising plot
[18-02-25 17:25:35 :INFO    ] - Animating
[18-02-25 17:25:36 :INFO    ] - Saving plot to plot/data_era5_south_vas.mp4
[18-02-25 17:26:45 :INFO    ] - Produced plot/data_era5_north_uas.mp4
[18-02-25 17:26:45 :INFO    ] - Saving to plot/data_era5_south_zg250.mp4
[18-02-25 17:26:45 :INFO    ] - Inspecting data
[18-02-25 17:26:46 :INFO    ] - Initialising plot
[18-02-25 17:26:46 :INFO    ] - Animating
[18-02-25 17:26:47 :INFO    ] - Saving plot to plot/data_era5_south_zg250.mp4
[18-02-25 17:26:58 :INFO    ] - Produced plot/data_era5_north_vas.mp4
[18-02-25 17:26:58 :INFO    ] - Saving to plot/data_era5_south_zg500.mp4
[18-02-25 17:26:58 :INFO    ] - Inspecting data
[18-02-25 17:26:59 :INFO    ] - Initialising plot
[18-02-25 17:26:59 :INFO    ] - Animating
[18-02-25 17:27:00 :INFO    ] - Saving plot to plot/data_era5_south_zg500.mp4
[18-02-25 17:28:14 :INFO    ] - Produced plot/data_era5_south_psl.mp4
[18-02-25 17:28:14 :WARNING ] - Not overwriting existing: plot/data_osisaf_south_siconca.mp4
[18-02-25 17:28:32 :INFO    ] - Produced plot/data_era5_south_ta500.mp4
[18-02-25 17:28:34 :INFO    ] - Produced plot/data_era5_south_tas.mp4
[18-02-25 17:28:44 :INFO    ] - Produced plot/data_era5_south_tos.mp4
[18-02-25 17:29:58 :INFO    ] - Produced plot/data_era5_south_zg250.mp4
[18-02-25 17:30:11 :INFO    ] - Produced plot/data_era5_south_uas.mp4
[18-02-25 17:30:11 :INFO    ] - Produced plot/data_era5_south_zg500.mp4
[18-02-25 17:30:15 :INFO    ] - Produced plot/data_era5_south_vas.mp4

Optionally, we can display the videos right here in the notebook:

from IPython.display import Video
Video("plot/data_era5_south_tas.mp4", embed=True, width=800)