What happened?

When computing the standard deviation over one dimension of a chunked DataArray, one may get NaNs.

What did you expect to happen?

We should not have any NaNs

Minimal Complete Verifiable Example

```Python x = (np.random.randn(10,10) + 1j*np.random.randn(10,10)) da = xr.DataArray(x).chunk(dict(dim_0=3))

da.std("dim_0").compute() # NaN da.compute().std("dim_0") # no NaNs

MVCE confirmation

• [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• [X] Complete example — the example is self-contained, including all data and the text of any traceback.
• [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

No response

Anything else we need to know?

No response

Environment

The following call in xarray/xarray/plot/plot.py does not seem to be thread safe: cbar = plt.colorbar(primitive, **cbar_kwargs) It leads to systematic crashes when distributed, with a cryptic error message (ValueError: Unknown element o). I have to call colorbars outside the xarray plot call to prevent crashes.

A call of the following type may fix the problem: cbar = fig.colorbar(primitive, **cbar_kwargs) But fig does not seem to be available directly in plot.py. Maybe: cbar = ax.get_figure().colorbar(primitive, **cbar_kwargs)

cheers

I generate a dataset once: ds = xr.DataArray(np.arange(10), name='x').to_dataset() ds.to_netcdf('test.nc', mode='w') Now I overwrite with a new netcdf file and load: ds = xr.DataArray(np.arange(20), name='x').to_dataset() ds.to_netcdf('test.nc', mode='w') ds_out = xr.open_dataset('test.nc') print(ds_out) outputs: <xarray.Dataset> Dimensions: (dim_0: 10) Dimensions without coordinates: dim_0 Data variables: x (dim_0) int64 ... I would have expected to get the new dataset.

If I use netcdf4, the file seems to have been properly overwritten: import netCDF4 as nc d = nc.Dataset('test.nc') d outputs: <class 'netCDF4._netCDF4.Dataset'> root group (NETCDF4 data model, file format HDF5): dimensions(sizes): dim_0(20) variables(dimensions): int64 x(dim_0) groups:

Environment:

• [X] follows #4046
• [X] Passes isort -rc . && black . && mypy . && flake8

I'm not sure the last point is really necessary for this PR, is it?

• [ ] Xref #1889
• [ ] Tests added
• [ ] Passes isort -rc . && black . && mypy . && flake8
• [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API

If you think this is relevant, I'll go ahead and start working on the items above, even though I'm not sure new tests are needed.

I'm in a situation where I want to generate a bidimensional DataArray from a method that takes each of the two dimensions as input parameters. I have two methods to do this but neither of these looks particularly elegant to me and I wondered whether somebody would have better ideas.

• Method 1 : dask delayed

``` x = np.arange(10) y = np.arange(20)

some_exp = lambda x, y: np.ones((Nstats)) some_exp_delayed = dask.delayed(some_exp, pure=True)

lazy_data = [some_exp_delayed(_x, _y) for _x in x for _y in y] sample = lazy_data[0].compute() arrays = [da.from_delayed(lazy_value, dtype=sample.dtype, shape=sample.shape) for lazy_value in lazy_data]

stack = (da.stack(arrays, axis=0) .reshape((len(x),len(y),sample.size)) ) ds = xr.DataArray(stack, dims=['x','y','stats']) ```

I tend to prefer this option because it imposes less requirement on the some_exp data shape. That being said it still seems like too many lines of code to achieve such result.

• Method 2: apply_ufunc

``` x = np.arange(10) y = np.arange(20)

ds = xr.Dataset(coords={'x': x, 'y': y}) ds['_y'] = (0*ds.x+ds.y) # breaks apply_ufunc otherwise ds = ds.chunk({'x': 1, 'y':1})

let's say each experiment outputs 5 statistical diagnostics

Nstats = 5 some_exp = lambda x, y: np.ones((1,1,Nstats))

out = (xr.apply_ufunc(some_exp, ds.x, ds._y, dask='parallelized', output_dtypes=[float], output_sizes={'stats': Nstats}, output_core_dims=[['stats']]) ) `` I don't understand why I have to use the dummy variableds._yin this case. Having to rely onapply_ufunc` seems like an overkill.

plot.line breaks depending on coordinate shape, see the code below:

```python x = np.arange(10) y = np.arange(20)

ds = xr.Dataset(coords={'x': x, 'y': y})

ds = ds.assign_coords(z=ds.y+ds.x) # goes through

ds = ds.assign_coords(z=ds.x+ds.y) # breaks ds['v'] = (ds.x+ds.y) ds['v'].plot.line(y='z', hue='x') This breaks with the following error: ... ~/.miniconda3/envs/equinox/lib/python3.7/site-packages/matplotlib/axes/_base.py in _plot_args(self, tup, kwargs) 340 341 if x.shape[0] != y.shape[0]: --> 342 raise ValueError(f"x and y must have same first dimension, but " 343 f"have shapes {x.shape} and {y.shape}") 344 if x.ndim > 2 or y.ndim > 2:

ValueError: x and y must have same first dimension, but have shapes (20, 10) and (10, 20) ```

I would have expected that that dimension order would not matter

Versions

Code Sample, a copy-pastable example if possible

This works fine: ```python nx, ny, nt = 32, 32, 64 ds = xr.Dataset({}, coords={'x':np.arange(nx),'y':np.arange(ny), 't': np.arange(nt)}) ds = ds.assign(v=ds.tnp.cos(np.pi/180./100ds.x)np.cos(np.pi/180./50ds.y)) ds = ds.chunk({'t': 1, 'x': nx/2, 'y': ny/2})

ds.isel(t=0).to_zarr('data_t0.zarr', mode='w') ```

But if I store, reload and select, I cannot store: ds.to_zarr('data.zarr', mode='w') ds = xr.open_zarr('data.zarr') ds.isel(t=0).to_zarr('data_t0.zarr', mode='w')

Error message ends with:

``` ~/.miniconda3/envs/equinox/lib/python3.6/site-packages/xarray/backends/zarr.py in _extract_zarr_variable_encoding(variable, raise_on_invalid) 181 182 chunks = _determine_zarr_chunks(encoding.get('chunks'), variable.chunks, --> 183 variable.ndim) 184 encoding['chunks'] = chunks 185 return encoding

~/.miniconda3/envs/equinox/lib/python3.6/site-packages/xarray/backends/zarr.py in _determine_zarr_chunks(enc_chunks, var_chunks, ndim) 112 raise ValueError("zarr chunks tuple %r must have same length as " 113 "variable.ndim %g" % --> 114 (enc_chunks_tuple, ndim)) 115 116 for x in enc_chunks_tuple:

ValueError: zarr chunks tuple (1, 16, 16) must have same length as variable.ndim 2 ```

Output of xr.show_versions()

Hi all,

We are trying to specify chunks after opening an mfdataset but it does not work. This works fine with datasets. Is this behavior expected? Are we doing anything wrong?

```

- Modules

import sys, os import xarray as xr

chunks = (1727, 2711) xr_chunks = {'x': chunks[-1], 'y': chunks[-2], 'time_counter':1, 'deptht': 1}

- Parameter

natl60_path = '/home7/pharos/othr/NATL60/' filename = natl60_path+'NATL60-MJM155-S/5d/2008/NATL60-MJM155_y2008m01d09.5d_gridT.nc' filenames = natl60_path+'NATL60-MJM155-S/5d/2008/NATL60-MJM155_y2008m01d0*gridT.nc'

dataset

open

ds = xr.open_dataset(filename,chunks=None)

chunk

ds = ds.chunk(xr_chunks)

plot

print 'With dataset:' print ds['votemper'].isel(time_counter=0,deptht=0).values

mfdataset

open

ds = xr.open_mfdataset(filenames,chunks=None, lock=False)

plot

print 'With mfdataset no chunks:' print ds['votemper'].isel(time_counter=0,deptht=0).values

chunk

print 'With mfdataset with chunks:' ds = ds.chunk(xr_chunks) print ds['votemper'].isel(time_counter=0,deptht=0) print ds['votemper'].isel(time_counter=0,deptht=0).values ```

The output is:

With dataset: [[ nan nan nan ..., nan nan nan] [ nan nan nan ..., nan nan nan] [ nan nan nan ..., nan nan nan] ..., [ nan nan nan ..., nan nan nan] [ nan nan nan ..., nan nan nan] [ nan nan nan ..., nan nan nan]] With mfdataset no chunks: [[ nan nan nan ..., nan nan nan] [ nan nan nan ..., nan nan nan] [ nan nan nan ..., nan nan nan] ..., [ nan nan nan ..., nan nan nan] [ nan nan nan ..., nan nan nan] [ nan nan nan ..., nan nan nan]] With mfdataset with chunks: <xarray.DataArray 'votemper' (y: 3454, x: 5422)> dask.array<getitem..., shape=(3454, 5422), dtype=float64, chunksize=(1727, 2711)> Coordinates: nav_lat (y, x) float32 26.5648 26.5648 26.5648 26.5648 26.5648 ... nav_lon (y, x) float32 -81.4512 -81.4346 -81.4179 -81.4012 ... deptht float32 0.480455 * x (x) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ... * y (y) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ... time_counter datetime64[ns] 2008-01-02T12:00:00 time_centered datetime64[ns] 2008-01-02T12:00:00 Attributes: long_name: temperature units: degC online_operation: average interval_operation: 40s interval_write: 5d

The code hangs for a while and then spits:

``` Traceback (most recent call last): File "/home/slyne/aponte/natl60/python/natl60_dimup/overview/aurelien/plot_snapshot_2d_v4_break.py", line 44, in <module> print ds['votemper'].isel(time_counter=0,deptht=0).values File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/xarray/core/dataarray.py", line 364, in values return self.variable.values File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/xarray/core/variable.py", line 288, in values return _as_array_or_item(self._data_cached()) File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/xarray/core/variable.py", line 254, in _data_cached self._data = np.asarray(self._data) File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/numpy/core/numeric.py", line 460, in asarray return array(a, dtype, copy=False, order=order) File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/array/core.py", line 867, in array x = self.compute() File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/base.py", line 37, in compute return compute(self, kwargs)[0] File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/base.py", line 110, in compute results = get(dsk, keys, kwargs) File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/threaded.py", line 57, in get **kwargs) File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/async.py", line 481, in get_async raise(remote_exception(res, tb)) dask.async.MemoryError:

Traceback

File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/async.py", line 264, in execute_task result = _execute_task(task, data) File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/async.py", line 245, in _execute_task args2 = [_execute_task(a, cache) for a in args] File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/async.py", line 245, in _execute_task args2 = [_execute_task(a, cache) for a in args] File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/async.py", line 242, in _execute_task return [_execute_task(a, cache) for a in arg] File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/async.py", line 242, in _execute_task return [_execute_task(a, cache) for a in arg] File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/async.py", line 242, in _execute_task return [_execute_task(a, cache) for a in arg] File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/async.py", line 242, in _execute_task return [_execute_task(a, cache) for a in arg] File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/async.py", line 245, in _execute_task args2 = [_execute_task(a, cache) for a in args] File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/async.py", line 246, in _execute_task return func(*args2) File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/dask/array/core.py", line 50, in getarray c = np.asarray(c) File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/numpy/core/numeric.py", line 460, in asarray return array(a, dtype, copy=False, order=order) File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/xarray/core/indexing.py", line 312, in array return np.asarray(array[self.key], dtype=None) File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/xarray/conventions.py", line 359, in getitem self.scale_factor, self.add_offset, self._dtype) File "/home1/homedir5/perso/aponte/miniconda2/envs/natl60/lib/python2.7/site-packages/xarray/conventions.py", line 57, in mask_and_scale values = np.array(array, dtype=dtype, copy=True) ```

Cheers

Aurelien

isel significantly slows down a simple mean calculation:

python ds = xr.open_dataset(grid_dir_nc+'Depth.nc', chunks={'face':1}) print(ds) % time print(ds.Depth.mean().values) leads to: <xarray.Dataset> Dimensions: (face: 13, i: 4320, j: 4320) Coordinates: * i (i) int64 0 1 2 3 4 5 6 7 ... 4313 4314 4315 4316 4317 4318 4319 * j (j) int64 0 1 2 3 4 5 6 7 ... 4313 4314 4315 4316 4317 4318 4319 * face (face) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 Data variables: Depth (face, j, i) float32 dask.array<shape=(13, 4320, 4320), chunksize=(1, 4320, 4320)> 1935.0237 CPU times: user 241 ms, sys: 16.9 ms, total: 258 ms Wall time: 1.05 s

ds = xr.open_dataset(grid_dir_nc+'Depth.nc', chunks={'face':1}) ds = ds.isel(i=slice(None,None,4),j=slice(None,None,4)) % time print(ds.Depth.mean().values) leads to: 1935.0199 CPU times: user 9.43 s, sys: 819 ms, total: 10.3 s Wall time: 2min 57s

Is this expected behavior?

Output of xr.show_versions()

I am using latest xarray version (pip install https://github.com/pydata/xarray/archive/master.zip)

I am trying to store data to a netcdf file, and have issues:

Data is created according to:

```python import numpy as np import xarray as xr

i = np.arange(4320) j = np.arange(4320) face = np.arange(13) v = xr.DataArray(np.random.randn(face.size, j.size, i.size), \ coords={'i': i, 'j': j, 'face': face}, dims=['face','j','i']) ```

The following works: file_out = 'rand.nc' v.to_netcdf(file_out) there is a minor warning: /home1/datahome/aponte/.miniconda3/envs/equinox/lib/python3.6/site-packages/distributed/utils.py:128: RuntimeWarning: Couldn't detect a suitable IP address for reaching '8.8.8.8', defaulting to '127.0.0.1': [Errno 101] Network is unreachable % (host, default, e), RuntimeWarning)

But this does not work: file_out = '/home1/datawork/aponte/mit_tmp/rand.nc' v.to_netcdf(file_out)

with the following error message:

The following may of some use: (equinox) aponte@datarmor1:~/mit_equinox/sandbox> stat -f -L -c %T /home1/datawork/aponte/mit_tmp/ gpfs (equinox) aponte@datarmor1:~/mit_equinox/sandbox> stat -f -L -c %T . nfs (the .directory being where the notebook seats)

Output of xr.show_versions()