issue_comments

OK, great. I'm going to close this then, and simply recommend that anyone encounter this issue try upgrading pandas.

@shoyer That did it. Under pandas 0.22, the DataArrays in aligned are all NaNs. I updated to pandas 0.23, and the non-NaN values were there as expected. To double-check, I downgraded to 0.22 again and got all NaNs again.

This what I see when printing aligned from your example: In [26]: aligned Out[26]: (<xarray.DataArray (x: 20, y: 5)> array([[ 0., nan, nan, nan, nan], [nan, 1., nan, nan, nan], [nan, nan, 2., nan, nan], [nan, nan, nan, 3., nan], [nan, nan, nan, nan, 4.], [ 5., nan, nan, nan, nan], [nan, 6., nan, nan, nan], [nan, nan, 7., nan, nan], [nan, nan, nan, 8., nan], [nan, nan, nan, nan, 9.], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan]]) Coordinates: * x (x) MultiIndex - ints (x) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 - nans (x) float64 nan nan nan nan nan nan nan nan nan nan nan nan nan ... * y (y) object 'a' 'b' 'c' 'd' 'e', <xarray.DataArray (x: 20, y: 5)> array([[nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [ 0., nan, nan, nan, nan], [nan, 1., nan, nan, nan], [nan, nan, 2., nan, nan], [nan, nan, nan, 3., nan], [nan, nan, nan, nan, 4.], [ 5., nan, nan, nan, nan], [nan, 6., nan, nan, nan], [nan, nan, 7., nan, nan], [nan, nan, nan, 8., nan], [nan, nan, nan, nan, 9.]]) Coordinates: * x (x) MultiIndex - ints (x) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 - nans (x) float64 nan nan nan nan nan nan nan nan nan nan nan nan nan ... * y (y) object 'a' 'b' 'c' 'd' 'e')

The only material difference I can see in our environments is that I'm running pandas 0.23 and you're running pandas 0.22. Can you try updating pandas and see if that fixes the issue?

I found a way to reproduce the error. One of the MuliIndex levels on the DataArrays has NaNs in it. If I remove that level, the correct values appear in the result. Should the presence of that MultiIndex level cause this behavior?

``` import string import numpy as np import xarray as xr

dims = ("x", "y") shape = (10, 5) das = [] for j in (0, 1): data = np.full(shape, np.nan, dtype="float64") for i in range(shape[0]): data[i, i % shape[1]] = float(i) coords_d = { "ints": ("x", range(jshape[0], (j+1)shape[0])), "nans": ("x", np.array([np.nan] * shape[0], dtype="float64")), "lower": ("y", list(string.ascii_lowercase[:shape[1]])) } da = xr.DataArray(data=data, dims=dims, coords=coords_d) da.set_index(append=True, inplace=True, x=["ints", "nans"], y=["lower"]) das.append(da) nonzeros_raw = [np.nonzero(~np.isnan(da)) for da in das] print("nonzeros_raw: ") print(nonzeros_raw) aligned = xr.align(*das, join="outer") nonzeros_aligned = [np.nonzero(~np.isnan(da)) for da in aligned] print("nonzeros_aligned: ") print(nonzeros_aligned) assert nonzeros_raw[0].shape == nonzeros_aligned[0].shape ```

This is what I would expect to see returned by align():

(<xarray.DataArray (x: 20, y: 5)> array([[ 0., nan, nan, nan, nan], [nan, 1., nan, nan, nan], [nan, nan, 2., nan, nan], [nan, nan, nan, 3., nan], [nan, nan, nan, nan, 4.], [ 5., nan, nan, nan, nan], [nan, 6., nan, nan, nan], [nan, nan, 7., nan, nan], [nan, nan, nan, 8., nan], [nan, nan, nan, nan, 9.], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan]]) Coordinates: * x (x) MultiIndex - ints (x) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 - nans (x) object nan nan nan nan nan nan nan nan nan nan nan nan nan ... * y (y) object 'a' 'b' 'c' 'd' 'e', <xarray.DataArray (x: 20, y: 5)> array([[nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [ 0., nan, nan, nan, nan], [nan, 1., nan, nan, nan], [nan, nan, 2., nan, nan], [nan, nan, nan, 3., nan], [nan, nan, nan, nan, 4.], [ 5., nan, nan, nan, nan], [nan, 6., nan, nan, nan], [nan, nan, 7., nan, nan], [nan, nan, nan, 8., nan], [nan, nan, nan, nan, 9.]]) Coordinates: * x (x) MultiIndex - ints (x) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 - nans (x) object nan nan nan nan nan nan nan nan nan nan nan nan nan ... * y (y) object 'a' 'b' 'c' 'd' 'e')

I see something very similar, but with the nans level removed, if I do this:
xr.align(*[da.reset_index("nans", drop=True) for da in das], join="outer")

Since the align is an outer join, I would expect all the non-NaN values in the original DataArrays to also appear in the aligned DataArrays.

Sorry, I'm not quite following -- can we please give a specific example of which output from your example looks wrong, and print how it should look instead?

For clarity, here are the prints of the arrays before and after alignment:

Before alignment:
[<xarray.DataArray (x: 10, y: 5)> array([[ 0., nan, nan, nan, nan], [nan, 1., nan, nan, nan], [nan, nan, 2., nan, nan], [nan, nan, nan, 3., nan], [nan, nan, nan, nan, 4.], [ 5., nan, nan, nan, nan], [nan, 6., nan, nan, nan], [nan, nan, 7., nan, nan], [nan, nan, nan, 8., nan], [nan, nan, nan, nan, 9.]]) Coordinates: * x (x) MultiIndex - ints (x) int64 0 1 2 3 4 5 6 7 8 9 - nans (x) float64 nan nan nan nan nan nan nan nan nan nan * y (y) object 'a' 'b' 'c' 'd' 'e', <xarray.DataArray (x: 10, y: 5)> array([[ 0., nan, nan, nan, nan], [nan, 1., nan, nan, nan], [nan, nan, 2., nan, nan], [nan, nan, nan, 3., nan], [nan, nan, nan, nan, 4.], [ 5., nan, nan, nan, nan], [nan, 6., nan, nan, nan], [nan, nan, 7., nan, nan], [nan, nan, nan, 8., nan], [nan, nan, nan, nan, 9.]]) Coordinates: * x (x) MultiIndex - ints (x) int64 10 11 12 13 14 15 16 17 18 19 - nans (x) float64 nan nan nan nan nan nan nan nan nan nan * y (y) object 'a' 'b' 'c' 'd' 'e']

After alignment:
(<xarray.DataArray (x: 20, y: 5)> array([[nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan]]) Coordinates: * x (x) MultiIndex - ints (x) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 - nans (x) object nan nan nan nan nan nan nan nan nan nan nan nan nan ... * y (y) object 'a' 'b' 'c' 'd' 'e', <xarray.DataArray (x: 20, y: 5)> array([[nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan], [nan, nan, nan, nan, nan]]) Coordinates: * x (x) MultiIndex - ints (x) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 - nans (x) object nan nan nan nan nan nan nan nan nan nan nan nan nan ... * y (y) object 'a' 'b' 'c' 'd' 'e')

Since the align is an outer join, I would expect all the non-NaN values in the original DataArrays to also appear in the aligned DataArrays. Perhaps I am misinterpreting the behavior of join="outer".

Thanks for the example. Can you please identify exactly which behavior you find surprising, and what you think the result should be?

Are you sure the indexes along the aligned dimensions match exactly? Small differences in floats are the most common source of this issue.

Try using second.reindex_like(first, method='nearest') instead of xarray.align(first, second).