issue_comments: 401728326

@rabernat thanks so far for all the help. So if pickle is not the way forward then I need to resize casts so they all have same dimensions. So I came up with the following code:

``` def expand_levels(dataset,maxlevel=1500): newds = xr.Dataset() blankstack = np.empty((dataset.N_PROF.size,maxlevel-dataset.N_LEVELS.size)) blankstack[:] = np.nan newds['N_PROF'] = dataset.N_PROF.values; newds['N_LEVELS'] = np.arange(maxlevel).astype('int64') newds['N_PARAM'] = dataset.N_PARAM newds['N_CALIB'] = dataset.N_CALIB for varname, da in dataset.data_vars.items(): if 'N_PROF' in da.dims: if 'N_LEVELS' in da.dims: newds[varname] = xr.DataArray(np.hstack((da.values, blankstack)), dims=da.dims, name=da.name, attrs=da.attrs) elif 'N_HISTORY' not in da.dims: newds[varname] = da newds.attrs = dataset.attrs return newds

def append_to_zarr(dataset,zarrfile): for varname, da in dataset.data_vars.items(): zarrfile[varname].append(da.values)

files =list(glob.iglob(r'D:\argo\csiro**_prof.nc', recursive=True)) expand_levels(xr.open_dataset(files[0]),3000).to_zarr(r'D:\argo\argo.zarr',mode='w') za =zarr.open(r'D:\argo\argo.zarr',mode='w+') for f in files[1:]: print(f) append_to_zarr(expand_levels(xr.open_dataset(f), 3000),za)

``` This basically appends nan on the end of the profiles to get them all the same length. Then I append them into the zarr structure. This is very experimental I just wanted to see how appending them all to big arrays would work. It might be better to save a resized netcdf and then open them all at once and do a to_zarr?