issue_comments

@shoyer That did it. Under pandas 0.22, the DataArrays in aligned are all NaNs. I updated to pandas 0.23, and the non-NaN values were there as expected. To double-check, I downgraded to 0.22 again and got all NaNs again.

I found a way to reproduce the error. One of the MuliIndex levels on the DataArrays has NaNs in it. If I remove that level, the correct values appear in the result. Should the presence of that MultiIndex level cause this behavior?

``` import string import numpy as np import xarray as xr

dims = ("x", "y") shape = (10, 5) das = [] for j in (0, 1): data = np.full(shape, np.nan, dtype="float64") for i in range(shape[0]): data[i, i % shape[1]] = float(i) coords_d = { "ints": ("x", range(jshape[0], (j+1)shape[0])), "nans": ("x", np.array([np.nan] * shape[0], dtype="float64")), "lower": ("y", list(string.ascii_lowercase[:shape[1]])) } da = xr.DataArray(data=data, dims=dims, coords=coords_d) da.set_index(append=True, inplace=True, x=["ints", "nans"], y=["lower"]) das.append(da) nonzeros_raw = [np.nonzero(~np.isnan(da)) for da in das] print("nonzeros_raw: ") print(nonzeros_raw) aligned = xr.align(*das, join="outer") nonzeros_aligned = [np.nonzero(~np.isnan(da)) for da in aligned] print("nonzeros_aligned: ") print(nonzeros_aligned) assert nonzeros_raw[0].shape == nonzeros_aligned[0].shape ```

This is what I would expect to see returned by align():

(<xarray.DataArray (x: 20, y: 5)> array([[ 0., nan, nan, nan, nan], [nan, 1., nan, nan, nan], [nan, nan, 2., nan, nan], [nan, nan, nan, 3., nan], [nan, nan, nan, nan, 4.], [ 5., nan, nan, nan, nan], [nan, 6., nan, nan, nan], [nan, nan, 7., nan, nan], [nan, nan, nan, 8., nan], [nan, nan, nan, nan, 9.], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan]]) Coordinates: * x (x) MultiIndex - ints (x) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 - nans (x) object nan nan nan nan nan nan nan nan nan nan nan nan nan ... * y (y) object 'a' 'b' 'c' 'd' 'e', <xarray.DataArray (x: 20, y: 5)> array([[nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [ 0., nan, nan, nan, nan], [nan, 1., nan, nan, nan], [nan, nan, 2., nan, nan], [nan, nan, nan, 3., nan], [nan, nan, nan, nan, 4.], [ 5., nan, nan, nan, nan], [nan, 6., nan, nan, nan], [nan, nan, 7., nan, nan], [nan, nan, nan, 8., nan], [nan, nan, nan, nan, 9.]]) Coordinates: * x (x) MultiIndex - ints (x) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 - nans (x) object nan nan nan nan nan nan nan nan nan nan nan nan nan ... * y (y) object 'a' 'b' 'c' 'd' 'e')

I see something very similar, but with the nans level removed, if I do this:
xr.align(*[da.reset_index("nans", drop=True) for da in das], join="outer")

For clarity, here are the prints of the arrays before and after alignment:

Before alignment:
[<xarray.DataArray (x: 10, y: 5)> array([[ 0., nan, nan, nan, nan], [nan, 1., nan, nan, nan], [nan, nan, 2., nan, nan], [nan, nan, nan, 3., nan], [nan, nan, nan, nan, 4.], [ 5., nan, nan, nan, nan], [nan, 6., nan, nan, nan], [nan, nan, 7., nan, nan], [nan, nan, nan, 8., nan], [nan, nan, nan, nan, 9.]]) Coordinates: * x (x) MultiIndex - ints (x) int64 0 1 2 3 4 5 6 7 8 9 - nans (x) float64 nan nan nan nan nan nan nan nan nan nan * y (y) object 'a' 'b' 'c' 'd' 'e', <xarray.DataArray (x: 10, y: 5)> array([[ 0., nan, nan, nan, nan], [nan, 1., nan, nan, nan], [nan, nan, 2., nan, nan], [nan, nan, nan, 3., nan], [nan, nan, nan, nan, 4.], [ 5., nan, nan, nan, nan], [nan, 6., nan, nan, nan], [nan, nan, 7., nan, nan], [nan, nan, nan, 8., nan], [nan, nan, nan, nan, 9.]]) Coordinates: * x (x) MultiIndex - ints (x) int64 10 11 12 13 14 15 16 17 18 19 - nans (x) float64 nan nan nan nan nan nan nan nan nan nan * y (y) object 'a' 'b' 'c' 'd' 'e']

After alignment:
(<xarray.DataArray (x: 20, y: 5)> array([[nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan]]) Coordinates: * x (x) MultiIndex - ints (x) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 - nans (x) object nan nan nan nan nan nan nan nan nan nan nan nan nan ... * y (y) object 'a' 'b' 'c' 'd' 'e', <xarray.DataArray (x: 20, y: 5)> array([[nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan]]) Coordinates: * x (x) MultiIndex - ints (x) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 - nans (x) object nan nan nan nan nan nan nan nan nan nan nan nan nan ... * y (y) object 'a' 'b' 'c' 'd' 'e')

Since the align is an outer join, I would expect all the non-NaN values in the original DataArrays to also appear in the aligned DataArrays. Perhaps I am misinterpreting the behavior of join="outer".

Will the new API preserve the order of the levels? One of the features that's necessary for MultiIndex to be truly hierarchical is that there is a defined order to the levels.

Is use case 1 (Multiple groupby arguments along a single dimension) being held back for use case 2 (Multiple groupby arguments along different dimensions)? Use case 1 would be very useful by itself.

In case anyone gets here by Googling something like "xarray groupby slow" and you loaded data from a netCDF file, be aware that slowness you see in groupby aggregation on a Dataset or DataArray may actually be due not to this issue but to the lazy loading that's done by default. This can be fixed by calling .load() on the Dataset or DataArray. See the Tip about lazy loading at http://xarray.pydata.org/en/stable/io.html#netcdf.

Makes sense. Just needs a doc update, then.

What's the preferred means to contribute doc (including little edits like this)? Pull requests?