issue_comments

I think I ran into a similar problem when combining dask-chunked DataSets (originating from open_mfdataset) with Python's native multiprocessing package. I get no error message, and the headers of the files are created, but then the script hangs indefinitely. The use case is: combining and resampling of variables into ~1000 different NetCDF files, which I want to distribute over different processes using multiprocessing.

MCVE Code Sample ```python import xarray as xr from multiprocessing import Pool import os

if (False): """ Load data without using dask """ ds = xr.open_dataset("http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis/surface/air.sig995.1960.nc") else: """ Load data using dask """ ds = xr.open_dataset("http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis/surface/air.sig995.1960.nc", chunks={})

print(ds.nbytes / 1e6, 'MB')

print('chunks', ds.air.chunks) # chunks is empty without dask

outdir = '/glade/scratch/lvank' # change this to some temporary directory on your system

def do_work(n): print(n) ds.to_netcdf(os.path.join(outdir, f'{n}.nc'))

tasks = range(10)

with Pool(processes=2) as pool: pool.map(do_work, tasks)

print('done') ```

Expected Output The NetCDF copies in outdir named 0.nc to 9.nc should be created for both cases (with and without Dask).

Problem Description In the case with Dask, when the if-statement evaluates to False, the files are not created and the program hangs.

Output of xr.show_versions()

Unfortunately, join='override' raises an IndexError.

Thanks Tom. This indeed gives a dataset with the correct dimensions but there is missing data

I've also tried join='override' but this raises an IndexError.

good point,np.array_equal(ds1.lat , ds2.lat) yields False whereas np.allclose() reported True.

How to use compat='override' in this case? I tried ds3 = xr.concat((ds1,ds2), dim='time',compat='override',coords='minimal') But this didn't work.

Thanks, I've implemented your suggestion as a workaround, but it fails with the following error:

NameError: name '_process_cmap_cbar_kwargs' is not defined

Stephan, thanks a lot for your code snippet from December, this is an elegant solution to the problem. One minor correction though, because I found that it fails to infer the period if none is given. The divide should be a multiplication I believe, i.e.

```python import xarray import numpy as np

def add_cyclic_point(xarray_obj, dim, period=None): if period is None: period = xarray_obj.sizes[dim] * xarray_obj.coords[dim][:2].diff(dim).item() first_point = xarray_obj.isel({dim: slice(1)}) first_point.coords[dim] = first_point.coords[dim]+period return xarray.concat([xarray_obj, first_point], dim=dim) ```

Any update on this issue? It would be great if add_cyclic_point could be applied to all variables automatically.

Just for other peoples reference, I now have this workaround, creating erai_jja_cy a 'cyclic' version of erai_jja: python dd, ll = add_cyclic_point(erai_jja.values, erai_jja.lon) erai_jja_cy = xr.DataArray(dd, coords={'lat':erai_jja.lat, 'lon':ll}, dims=('lat','lon'))

thanks for your elaborate response @spencerkclark

Do you happen to be using a PeriodIndex because of pandas Timestamp-limitations?

Yes, the main limitation being the limited range of years (~584) whereas my dataset spans 1800 years. Note that in glaciology, which deals with ice sheet responses over multiple millennia, this is considered a short period.

I elaborated a bit more on my problem in this issue which is in a unofficial repo, I realized too late.

Anyway, your code using cftime solves my problem 😄 indeed resampling to 'AS-JUN' is what I was looking for. Still, it would be nice to have better support for PeriodIndex in the future. It has costed me a lot of time figuring out what's going on and learning the details of all the different date & time implementations. Which is a waste in the end.

+1 to this issue. I'm struggling big time with an 1800-year climate model dataset that I need to resample in order to make different annual means (June-May).

I stumbled across the same problem in xarray 0.9.1 and updating to 0.10.2 solved it. Perhaps this issue may be closed?